> Two years ago, Facebook open-sourced Zstandard v1.0...<p>Bullshit, Zstd was open-source from the very beginning, they just hired Yann and moved the project under facebook org. How do I know? I have written the JVM bindings [1] since v0.1 that are now used by Spark, Kafka, etc.<p>EDIT: Actually, my initial bindings were against v0.0.2 [2]<p>Kudos to FB for hiring him and helping Zstd getting production ready. This is just a PR false claim.<p>[1] <a href="https://github.com/luben/zstd-jni" rel="nofollow">https://github.com/luben/zstd-jni</a><p>[2] <a href="https://github.com/luben/zstd-jni/commit/3dfe760cbb8cc46da3268af6aa73dce6014298ef" rel="nofollow">https://github.com/luben/zstd-jni/commit/3dfe760cbb8cc46da32...</a>
My browser loaded that website with a header:
accept-encoding: gzip, deflate, br ("br" means Brotli by Google)<p>The response had a header:
content-encoding: gzip<p>Zstandard looks like an improvement of DEFLATE (= gzip = zlib) and its specification is only 3x longer (even though it is introduced 22 years later): <a href="https://tools.ietf.org/html/rfc8478" rel="nofollow">https://tools.ietf.org/html/rfc8478</a><p>Since Zstandard is so simple and efficient, I thought it would get into browsers very quickly. Then, it could make sense to compress even PNG or JPG images, which are usually impossible to compress with DEFLATE.
I'd really like to thank Cyan for their contributions. `zstd` and `lz4` are great. I'm pretty much exclusively using `zstd` for my tarball needs in the present day as it beats the pants off `gzip` and for plane text code (most of what I compress) it performs amazingly. (shameless self promotion) I wrote my own tar clone to make usage of it [1].<p>It is nice to have disk IO be the limiting factor on decompression even when you are using NVMe drives.<p>[1] <a href="https://github.com/valarauca/car" rel="nofollow">https://github.com/valarauca/car</a>
The best thing about zstd is its zlibWrapper, which lets you write code as if you’re consuming zlib-compressed files while transparently working with zlib-, zstd-, or uncompressed files. I build several of my tools with zstd for this reason.
I used zstandard to compress mesages in P2P multiplayer game engine, and, taught on our real-life packets, it got us 2x-5x improvement. Awesome library, will use it in any similar project from now on.
> And the zstd binary itself includes a dictionary trainer (zstd --train). Building a pipeline for handling compression dictionaries can therefore be reduced to being a matter of gluing these building blocks together in relatively straightforward ways.<p>What happens if your user data trained dictionary ends up storing user data and you receive a GPDR destruction request?
I find the first chart so hard to understand. The axes need labels, and the color scheme is not ideal. They should use different line styles, and add a caption below summarizing the findings. There's a reason journals often require graphs to be formatted this way.<p>This is a resource I've found helpful:
<a href="https://www3.nd.edu/~pkamat/pdf/graphs.pdf" rel="nofollow">https://www3.nd.edu/~pkamat/pdf/graphs.pdf</a><p>"Consider readers with color blindness
or deficiencies"<p>"Avoid colors that are difficult to distinguish"
I hope they pay greater attention to the low and high end of their compression ratio spectrum. On the low end, it'd be great if it could exceed lz4 in terms of speed and memory savings. On the high end it'd be great to exceed XZ/LZMA.<p>Right now it's impressive "in the middle", but I find myself in a lot of situations where I care about the extremes. I.e. for something that will be transferred a lot, or cold-stored, I want maximum compression, CPU/RAM usage be damned, within reason. So I tend to use LZMA there if files aren't too large. For realtime/network RPC scenarios I want minimum RAM/CPU usage and Pareto-optimality on multi-GbE networks. This is where I use LZ4 (and used to use Snappy/Zippy).<p>At their scale, though, FB is surely saving many millions of dollars thanks to deploying this, both in human/machine time savings and storage savings.