It's basically byte or bit shuffling filter (very fast SIMD optimized) in front of several modern compressors (lz4, zstd, their own) with self describing header. So if you have an array of 100 8-byte values, the result of shuffling is 100 1st bytes, followed by 100 of 2nd bytes and so on.<p>It shines when values are of fixed size with lots of similar bits, e.g. positive integers of the same magnitude. It's not so good for doubles, where bits change a lot. Also, if stroring diffs it helps to take a diff from initial value in a chunk, not previous value, so that deltas change sign less often (and most bits flipped).<p>From own usage case, for the same data, C# decimal (16 bytes struct) is compressed much better than doubles (final absolute blob size), while decimal is taking 2x more memory uncompressed.<p>If data items have little similar bits/bytes then it's underlying compressor that matters.
Back when I did HPC work, I used Blosc to compress information about atoms for molecular dynamics simulations before transferring this data between the Infiniband interconnects. Despite the high speed of the interconnects, it was actually faster to compress, transmit, and decompress using Blosc than to transmit only the raw data.
Blosc is an outstanding project. I have used it with great success in finance and general data science in production with very large total datasets (one custom binary format and one leveraging protobufs).<p>It really shines first and foremost as a meta compressor, giving the developer a clean block based API. Once integrated (which really is quite easy) you can experiment easy with different compressors and preconditioners to see what works best with your dataset. These things can be changed at runtime and give you great flexibility.<p>Francesc has been advancing blosc consistently with a steady vision for years and years. It is one of the most underrated tools around IMO.
Apparently they have several benchmarks where they claim that decompression is faster than memcpy (!).<p>However, this is only the case because on several Intel x86_64 benchmarks they report memcpy performance between 5-10 GB/s, while even a basic DDR3 dual channel arch has 20 GB/s memory bandwidth, while a modern quad channel DDR4 can have 76.8 GB/s bandwidth, and of course there is no reason for memcpy to be substantially slower than memory bandwidth assuming it's properly implemented (AVX can separately read two and write one 256-bit per cycle = 128 GB/s memcpy at 4GHz).<p>Am I missing something or is this another case of "implausible claims = they screwed the benchmark = they are incompetent/malicious"?
It's very good! I have used Blosc in developing JDF.jl a serialization format for dataframes.<p><a href="https://github.com/xiaodaigh/JDF.jl" rel="nofollow">https://github.com/xiaodaigh/JDF.jl</a>
The used shuffle techniques before compresson might be useful for squashfs? We play around with a mesh network (freifunk.net) and there are ton's of cheap 4mb flash devices that need every kb of storage :)
Blosc is an excellent choice if speed is what you are after. Give or take 5 years ago I had to use a compression to transport a lot of data over zmq and blosc ran in circles over all other compressions.
Can someone with Blosc 2 experience tell me what are the proper conditions to use superchunks or frames? When does it become advantageous to use one over the other?<p>This is a really interesting library.
This would be an excellent candidate to put on an FPGA directly next to the CPU. (Assuming such thing would exist and be open enough to be usable by general public.)