TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

Ebiggers/libdeflate: Heavily optimized DEFLATE/zlib/gzip library

3 pointsby xioxoxover 1 year ago

3 comments

powturboover 1 year ago
You can use TurboBench [1] to benchmark libdeflate against zlib, igzip, zlib-ng and others.<p>Download TurboBench from Releases [2]<p>Here Some Benchmarks:<p>- <a href="https:&#x2F;&#x2F;github.com&#x2F;zlib-ng&#x2F;zlib-ng&#x2F;issues&#x2F;1486">https:&#x2F;&#x2F;github.com&#x2F;zlib-ng&#x2F;zlib-ng&#x2F;issues&#x2F;1486</a><p>- <a href="https:&#x2F;&#x2F;github.com&#x2F;powturbo&#x2F;TurboBench&#x2F;issues&#x2F;43">https:&#x2F;&#x2F;github.com&#x2F;powturbo&#x2F;TurboBench&#x2F;issues&#x2F;43</a><p>[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;powturbo&#x2F;TurboBench">https:&#x2F;&#x2F;github.com&#x2F;powturbo&#x2F;TurboBench</a><p>[2] <a href="https:&#x2F;&#x2F;github.com&#x2F;powturbo&#x2F;TurboBench&#x2F;releases">https:&#x2F;&#x2F;github.com&#x2F;powturbo&#x2F;TurboBench&#x2F;releases</a>
mxmlnknover 1 year ago
Based on my benchmarks, ISA-l&#x2F;igzip is more than twice(!) as fast as libdeflate and zlib for decompression. I&#x27;m almost enamored with ISA-l because of its speed. And yes, it works on AMD and also has Assembler code for ARM, so probably also works on ARM.<p>For parallelized decompression of gzip, I recommended my own tool, rapidgzip. I have measured up to 10 GB&#x2F;s decompression bandwidth with it (&gt;20 GB&#x2F;s if an index already exists). I&#x27;m currently working on integrating ISA-l for even more special cases into rapidgzip and hope to release version 0.9.0 in the next days. It will have another +30-100% performance boost for many cases, thanks to ISA-l.
评论 #37318528 未加载
评论 #37333432 未加载
xioxoxover 1 year ago
One nice thing about this is that the compression factor seems quite a bit higher than standard gzip. This is very useful if you need compatibility with gzip&#x2F;zlib formats (e.g. PNG files). For my use case, the compressed output was 8% smaller using libdeflate compression=12 than with gzip compression=9. The compression was also a lot faster with a single thread. It would be nice to see a comparison with a larger number of inputs, however.