>"Also, consider that if you can only read at 100 MB/s off a mechanical drive but your CPU can decompress data at ~500 MB/s then the mechanical drive is able to provide 5x the throughput you'd otherwise expect thanks to compression."<p>I'd not really thought of that aspect before... My old brain is hard-coded to save cpu cycles ... Time to change my ways :)
Lossless compression is apparently equivalent to general intelligence:<p><a href="http://mattmahoney.net/dc/rationale.html" rel="nofollow">http://mattmahoney.net/dc/rationale.html</a>
Doesn't LZFSE stem from the Finite State Entropy library Yann Collet (LZ4, ZSTD) wrote as a base for ZSTD (together with HUFF0) and Apple just decided to use it before it was fully mature? So shouldn't LZFSE be a predecessor to ZSTD in the Tree?
Could anyone explain why LZ77 is preferred by implementors versus LZ78?<p>It seems important for compressibility to prepare the data for maximum self-similarity, in addition to the LZ algorithms (as evidenced by the sort in this article). Could someone point towards a good modern summary of the approaches or heuristics?
>Entropy, an Information Theory term coined by Claude Shannon in 1948, describes the minimum number of bits, on average, needed to encode a dataset.<p>Shannon didn't coin the term entropy. He borrowed it from the analogous definition in thermodynamics.