<a href="https://github.com/facebook/zstd/issues/395#issuecomment-535875379" rel="nofollow">https://github.com/facebook/zstd/issues/395#issuecomment-535...</a><p>If anyone from Facebook is reading, please look at making zstd seekable. There is already experimental support but it needs to be standardized. This could open up really interesting new use-cases [see my comment linked above], where currently we have to use much slower xz/lzma instead.
I have been incredibly impressed with Zstandard, and if you are not currently using I suggest giving it a try.<p>I'm a sysadmin so this has worked it's way into every corner of my servers, from backups, to log compression, and more. Most of my servers run FreeBSD, and with the merge of linux/FreeBSD ZFS I am looking forward to the integration of ZSTD into ZFS so I can use that for on-the-fly Zpool compression.<p>I can safely say that ZSTD has saved me TB of space over LZ4 & GZIP.
Zstandard is great indeed. Performance is excellent, but in my experience its killer feature is the user definable dictionary. My search engine software stores its documents in an archive to which I need random access. Lots of small documents, random access: bad for compression. Use of zstd with a tuned dictionary improved compression fivefold(!) when compared to the older zlib solution.
ZStandard is great - I used it as a replacement for PNG
for texture loading in the game engine for Weapon Hacker. Also for all game entity data - reduced asset loading time by about 4x, so things pretty much start instantly.
Zstandard uses finite state entropy (FSE) for entropy encoding (after finding repeats), which in turn us based on asymmetric numeral system (ANS) encoding.<p>The canonical paper on ANS is indecipherable to me (i honestly think it's poorly written, it's not just me, although it is partly me!), but this seems like a clearer explanation (although i still don't entirely get it):<p><a href="https://bjlkeng.github.io/posts/lossless-compression-with-asymmetric-numeral-systems/" rel="nofollow">https://bjlkeng.github.io/posts/lossless-compression-with-as...</a>
For latency-sensitive operations - in particular, decompression - lz4 maintains an extreme lead in performance (at a cost of lower compression ratios). This has borne out in practice for me as well.
It would be good to see some charts[1] similar to the zfs zstd integration. At that time, I found it notable that LZ4 was still the most optimized point for anyone without an actual preference between speed and compression.<p>1 <a href="https://github.com/openzfs/zfs/pull/10278" rel="nofollow">https://github.com/openzfs/zfs/pull/10278</a>