LZ4 – Extremely fast compression

A while ago I did some simplistic SquashFS pack&#x2F;unpack benchmarks[1][2]. I was primarily interested in looking at the behavior of my thread-pool based packer, but as a side effect I got a comparison of compressor speed &amp; ratios over the various available compressors for my Debian test image.<p>I must say that LZ4 definitely stands out for both compression and uncompression speed, while still being able to cut the data size in half, making it probably quite suitable for life filesystems and network protocols. Particularly interesting was also comparing Zstd and LZ4[3], the former being substantially slower, but at the same time achieving a compression ratio somewhere between zlib and xz, while beating both in time (<i>in my benchmark</i> at least).<p>[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;AgentD&#x2F;squashfs-tools-ng&#x2F;blob&#x2F;master&#x2F;doc&#x2F;benchmark.txt" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;AgentD&#x2F;squashfs-tools-ng&#x2F;blob&#x2F;master&#x2F;doc&#x2F;...</a><p>[2] <a href="https:&#x2F;&#x2F;github.com&#x2F;AgentD&#x2F;squashfs-tools-ng&#x2F;blob&#x2F;master&#x2F;doc&#x2F;benchmark.ods" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;AgentD&#x2F;squashfs-tools-ng&#x2F;blob&#x2F;master&#x2F;doc&#x2F;...</a><p>[3] <a href="https:&#x2F;&#x2F;github.com&#x2F;AgentD&#x2F;squashfs-tools-ng&#x2F;blob&#x2F;master&#x2F;doc&#x2F;benchmark.txt#L233" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;AgentD&#x2F;squashfs-tools-ng&#x2F;blob&#x2F;master&#x2F;doc&#x2F;...</a>

LZ4 is so fast, that in make sense to use it everywhere over uncompressed data. Even storing items in-memory compressed sometimes is profitable as you can fit more items in memory.<p>Still zstd offers way better compression and got variable difficulty factor: <a href="https:&#x2F;&#x2F;github.com&#x2F;facebook&#x2F;zstd" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;facebook&#x2F;zstd</a> Decompression is always fast, but you can trade off compression vs. ratio factor.<p>In general if send data over network zstd is quite profitable. Even network attached disk AWS EBS or AWS S3 it can be a hugely profitable.

another contender is zstd: <a href="https:&#x2F;&#x2F;github.com&#x2F;facebook&#x2F;zstd" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;facebook&#x2F;zstd</a>. It typically offers better compression ratios than LZ4 at a slight (depending on your data) cost in speed. Additionally it offers a training mode to tune the algorithm to increase compression ratio on specific types of data, particularly useful for compression of small pieces of data.

I ported the block format to Rust matching the C implementation in performance and ratio.<p><a href="https:&#x2F;&#x2F;github.com&#x2F;pseitz&#x2F;lz4_flex" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;pseitz&#x2F;lz4_flex</a>

It is very interesting that compression libraries from Yann Collet outperform their Google counterparts by all means:<p>lz4 &gt;&gt; snappy<p>zstd &gt;&gt; brotli

I&#x27;m not a fan of the stacked bar charts, I like the table of data for &quot;Benchmarks&quot; on the github source page: <a href="https:&#x2F;&#x2F;github.com&#x2F;lz4&#x2F;lz4" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;lz4&#x2F;lz4</a><p>It makes it very clear where LZ4 fits into comparisons with compression speed, decompression speed and compression ratio

Here’s a fork of the Windows compression tool 7-Zip which has LZ4 support baked in along with some other useful algorithms – the repo has a good comparison of them: <a href="https:&#x2F;&#x2F;github.com&#x2F;mcmilk&#x2F;7-Zip-zstd&#x2F;" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;mcmilk&#x2F;7-Zip-zstd&#x2F;</a><p>(Linking to this more for the overview than the Windows tool in itself.)

LZ4 is so fast there’s almost no reason to NOT have it on for zfs volumes.

It is possible to make LZ4 decompression even faster, here&#x27;s how: <a href="https:&#x2F;&#x2F;habr.com&#x2F;en&#x2F;company&#x2F;yandex&#x2F;blog&#x2F;457612&#x2F;" rel="nofollow">https:&#x2F;&#x2F;habr.com&#x2F;en&#x2F;company&#x2F;yandex&#x2F;blog&#x2F;457612&#x2F;</a>

Some interesting and related projects:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;strigeus&#x2F;ipzip" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;strigeus&#x2F;ipzip</a> - TCP&#x2F;IP Packet Compressor with LZ4 support<p><a href="https:&#x2F;&#x2F;github.com&#x2F;centaurean&#x2F;density" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;centaurean&#x2F;density</a> - Extremely fast de&#x2F;compression

If I remember correctly, it is very popular in video games because it is faster to load compressed assets from disk and decompress them in memory than loading the uncompressed assets from disk, even on an SSD.

I&#x27;ve been hunting for a good decompressor to use in a low ram microcontroller, for instance, an ARM Cortex M0. I&#x27;ve read an article [1] on LZ4 decompression on the Cortex, but I couldn&#x27;t understand what kind of memory requirements are needed.<p>I&#x27;ve yet to really understand what kind of footprint LZ4 uses, and if it&#x27;s dependent on dictionary size used to compress. What if I have, say, 4KB that I could use to store in-place decompression. Is that related to the compression ratio?<p>[1] <a href="https:&#x2F;&#x2F;community.arm.com&#x2F;developer&#x2F;ip-products&#x2F;processors&#x2F;b&#x2F;processors-ip-blog&#x2F;posts&#x2F;lz4-decompression-routine-for-cortex-m0-and-later" rel="nofollow">https:&#x2F;&#x2F;community.arm.com&#x2F;developer&#x2F;ip-products&#x2F;processors&#x2F;b...</a>

<a href="https:&#x2F;&#x2F;blog.logrocket.com&#x2F;rust-compression-libraries&#x2F;" rel="nofollow">https:&#x2F;&#x2F;blog.logrocket.com&#x2F;rust-compression-libraries&#x2F;</a><p>Although implementations arein rust, I assume the provided benchmarks are representative of any optimised implementation...<p>Many compressor algorithms are compared on several data sets.<p>The results tables show compressed size, compression and decompression times for a number of normal and pathological cases.<p>Get a good feel about strengths and weaknesses.<p>Some algs really go downhill in pathological cases, such as with random data.<p>Do consider encryption too though you probably want to do that on the compressed data set where possible.<p>Sometimes external encryption means you will be stuck with something close to pathological...

LZ4 rocks. Used it in the past with great results (much less CPU intensive then the GZIP we were using, still getting good compression).

Weird that the page doesn&#x27;t list macOS or other Apple&#x27;s OS in the list of operating systems with LZ4 support.

If they&#x27;re including transfer time it&#x27;d be fun to see how raw data performs.

I&#x27;ve personally used LZ4 on production a scale that really proves how using a compression like LZ4 is more efficient than uncompressed data <a href="https:&#x2F;&#x2F;doordash.engineering&#x2F;2019&#x2F;01&#x2F;02&#x2F;speeding-up-redis-with-compression&#x2F;" rel="nofollow">https:&#x2F;&#x2F;doordash.engineering&#x2F;2019&#x2F;01&#x2F;02&#x2F;speeding-up-redis-wi...</a>

LZ4 is awesome, thank you for it. Another interesting and fast compressor I saw a while back was Density.

How does it work?

Interana uses lz4 for compressing certain data types, namely list-type data. We found it offers the best decompression speed for reasonably good compression ratio. We actually use the high compression variant.

Not sure how LZ4 compares but what convinced me to use brotli over some other compression libraries was how trivialy easy it was to integrate in C++ as well as in javascript. No effort at all. I wish more lib providers would make their things so approachable.

OK but how does it compare to Kraken?

The wikipedia article about LZ4 says that compression is slightly worse, compression speed is comparable, and decompression can be significantly slower compared to LZO. Can anyone enlighten me why I should not used LZO? I do not know anything about either algorithms, but if I switch from deflate I want to mak the right decision.