Scaling down off-the-shelf data compression: Backwards-compatible fine-grain mixing

Michael Gray, Peter Peterson, Peter Reiher

Research output: Chapter in Book/Report/Conference proceedingConference contribution

4 Scopus citations

Abstract

Pu and Singaravelu presented Fine-Grain Mixing, an adaptive compression system which aimed to maximize CPU and network utilization simultaneously by splitting a network stream into a mixture of compressed and uncompressed blocks. Blocks were compressed opportunistically in a send buffer; they compressed as many blocks as they could without becoming a bottleneck. They successfully utilized all available CPU and network bandwidth even on high speed connections. In addition, they noted much greater throughput than previous adaptive compression systems. Here, we take a different view of FG-Mixing than was taken by Pu and Singaravelu and give another explanation for its high performance: that fine-grain mixing of compressed and uncompressed blocks enables off-the-shelf compressors to scale down their degree of compression linearly with decreasing CPU usage. Exploring the scaling behavior in-depth allows us to make a variety of improvements to fine-grain mixed compression: better compression ratios for a given level of CPU consumption, a wider range of data reduction and CPU cost options, and parallelized compression to take advantage of multi-core CPUs. We make full compatibility with the ubiquitous deflate decompressor (as used in many network protocols directly, or as the back-end of the gzip and Zip formats) a primary goal, rather than using a special, incompatible protocol as in the original implementation of FG-Mixing. Moreover, we show that the benefits of fine-grain mixing are retained by our compatible version.

Original languageEnglish (US)
Title of host publicationProceedings - 32nd IEEE International Conference on Distributed Computing Systems, ICDCS 2012
Pages112-121
Number of pages10
DOIs
StatePublished - Oct 5 2012
Event32nd IEEE International Conference on Distributed Computing Systems, ICDCS 2012 - Macau, China
Duration: Jun 18 2012Jun 21 2012

Other

Other32nd IEEE International Conference on Distributed Computing Systems, ICDCS 2012
CountryChina
CityMacau
Period6/18/126/21/12

Fingerprint Dive into the research topics of 'Scaling down off-the-shelf data compression: Backwards-compatible fine-grain mixing'. Together they form a unique fingerprint.

  • Cite this

    Gray, M., Peterson, P., & Reiher, P. (2012). Scaling down off-the-shelf data compression: Backwards-compatible fine-grain mixing. In Proceedings - 32nd IEEE International Conference on Distributed Computing Systems, ICDCS 2012 (pp. 112-121). [6257984] https://doi.org/10.1109/ICDCS.2012.21