I wrote a tool, pixz, that does xz compression in parallel to take advantage of multiple cores. It also indexes xz-compressed tarballs, so you can extract an individual file very quickly instead of needing to decompress the entire tarball. The parallel-compressed, self-contained tarball+index is fully compatible with regular tar/xz, you don't need any special tools to extract it.<p><a href="http://github.com/vasi/pixz" rel="nofollow">http://github.com/vasi/pixz</a><p>The interface is still very rough, but it works. The xz utility comes with a very nice library and API, which made this a lot easier--thanks, Lasse!
The specification ( <a href="http://tukaani.org/xz/xz-file-format.txt" rel="nofollow">http://tukaani.org/xz/xz-file-format.txt</a> ) reads like this format allows the application of a chain of compression "filters". Perhaps the most interesting, and least portable is the filter that modifies code to make compression easier. (Section 5.2.3) Is this mainly for intel architecures?
The only reason I stick with gzip over bz2 or 7z (lzma) is that gzip is everywhere. In reality, nobody has a file compression utility that can handle LZMA (even though I do, I never use it).
I use 7-zip 9.15 beta - it supports LZMA2 compression and available to download here:<p><a href="http://www.7-zip.org/" rel="nofollow">http://www.7-zip.org/</a>
Xz appears to use the same algorithm as 7zip.<p>Here are some comparisons between the big 3 compression algorithms (taken from <a href="http://blogs.reucon.com/srt/tags/compression/" rel="nofollow">http://blogs.reucon.com/srt/tags/compression/</a> -- he used a 163 MB Mysql dump file for the tests):<p><pre><code> Compressor Size Ratio Compression Decompression
gzip 89 MB 54 % 0m 13s 0m 05s
bzip2 81 MB 49 % 1m 30s 0m 20s
7-zip 61 MB 37 % 1m 48s 0m 11s</code></pre>
While we're talking about compression algorithms, here is a nice little gem called BMZ:<p><a href="http://bitbucket.org/mattsta/bmz/src" rel="nofollow">http://bitbucket.org/mattsta/bmz/src</a><p>It's is a fast compression scheme implemented using BMDiff and a Google Zippy clone (based off LZO).
Tried to apt-get it (on Ubuntu 9.10) but got scared when I gotta a warning about lzma getting uninstalled and possibly breaking my dpkg.<p>Looking good on ubuntu 10.4<p>Same folder compressed in 1/2 the size as with tar cvfz, sick.