TSAC: Low Bitrate Audio Compression

Always love a good bellard ship.<p>This is definitely better than some of the others out there. I threw together some comparisons here at 7kb&#x2F;s for mp3&#x2F;opus&#x2F;aac: <a href="https:&#x2F;&#x2F;non.io&#x2F;TSAC-Comparisons" rel="nofollow">https:&#x2F;&#x2F;non.io&#x2F;TSAC-Comparisons</a><p>Happy to add other comparisons if others want any.<p>Overall, it&#x27;s FAR better at these lower bit rates, but that doesn&#x27;t mean it&#x27;s necessarily good. One issue I see off the bat is that volume is fairly inconsistent in the output for TSAC, which makes stereo in particular quite hard to listen to with the volume &quot;flickering&quot; in each channel independently.

if you patch out the CRC check in the binary with<p>echo -ne &quot;\x90\x90&quot; | dd if=&#x2F;dev&#x2F;stdin of=tsac bs=1 seek=23914 conv=notrunc<p>you can corrupt the compressed files with very interesting results: <a href="https:&#x2F;&#x2F;meow.social&#x2F;@mimir&#x2F;112238998609778334" rel="nofollow">https:&#x2F;&#x2F;meow.social&#x2F;@mimir&#x2F;112238998609778334</a><p>the fast mode (you don&#x27;t have to patch the binary for this one, it seems to not do the CRC check?) and the normal (non-fast) mode sound different, but both quite interesting

This doesn&#x27;t have much of a use case.<p>- Can&#x27;t use it in telephony (obvious application for low bitrates); phone handsets and headsets don&#x27;t have the power to do it in real time.<p>- Very small files of good quality would be useful in tiny embedded systems that have low flash space: but what systems of that type have the processing power for decoding? Very low storage more or less goes hand in hand with weak processing.<p>The quality is astonishing for the bit rate, though.

Clicked the download link wanting to take a look at the source... and was a bit perplexed before quickly canceling it. 237MB, <i>compressed</i>, for an audio codec!? At that point one can&#x27;t help but think that the samples are already in the decoder itself.<p>I wonder how it compares to <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Codec2" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Codec2</a> and related codecs, which go even lower for bitrate.

<i>An Nvidia GPU is necessary for fast operation.</i><p>Compression is getting so heavy that soon it isn&#x27;t possible to perform it on normal hardware. AV1 already proved that, the future audio&#x2F;video codecs will be even heavier.<p>Decompression is also getting heavier. Poor mobile devices.<p>I&#x27;m starting to appreciate well written algorithms which don&#x27;t require massive computing power. JPEG XL is a good example. It has the same compression ratio as AVIF, but requires less processing power.

One of the DAC authors here (the codec that this builds off of). Very cool work! Would love to see some more detail on the modifications to DAC. Boosting the capacity with a transformer makes sense to me.<p>Makes me happy to see DAC getting built on! Thanks!

I might be missing something obvious, but it&#x27;s not clear to me how to get an mp3 out of this on Ubuntu 22.04.<p>Following the docs, `.&#x2F;tsac c myfile.mp3 myfile.tsac` generates a tsac file that&#x27;s unplayable with mpv. Trying ffmpeg to convert to mp3 didn&#x27;t work: `ffmpeg -i myfile.tsac compressed.mp3` (&quot;myfile.tsac: Invalid data found when processing input&quot;). Using a wav input file has the same result.<p>I can use `.&#x2F;tsac d myfile.tsac output.wav` (I don&#x27;t really want to decompress anything, but worth a try) but then after compressing `output.wav` with `ffmpeg -i output.wav output.mp3`, output.mp3 is the same size as if I hadn&#x27;t used tsac (of course). If I use ffmpeg with a low bitrate like `-b:a 16k`, I get the usual low-quality gargle rather than the tsac output.

FYI (and in case Mr Bellard is reading), for the &quot;Greatest Love of All&quot; demo, the sample labeled &quot;mono 5.02 kb&#x2F;s&quot; is in fact linked to the 6.79 kb&#x2F;s stereo sample. The correct file is available at <a href="https:&#x2F;&#x2F;bellard.org&#x2F;tsac&#x2F;Greatest_Love_mono.wav" rel="nofollow">https:&#x2F;&#x2F;bellard.org&#x2F;tsac&#x2F;Greatest_Love_mono.wav</a>

This is quite similar to the models used by all the AI music generators. Some feed the tokens into a language model to generate music, some replace the tokenization part with an alternative that gives a continuous representation for diffusion models.

New advancement of media compression seems always focusing on low bitrate, be it audio, video or image.<p>Which is totally fair given their applications, but I always wonder how much improvement they bring in high bitrate scenario. For example, are there codecs that have much better (perceptible) quality than Apple AAC 256kbps (or achieving similar quality at, say, 160kbps?) How much better are AV1 at 10Mbps compared to H265&#x2F;264 (the improvement of H265 compared to H264 in &quot;transparent&quot; encoding was pretty disappointing IMHO).

&gt; The Transformer model is evaluated in a deterministic and reproducible way. Hence the result does not depend on the exact GPU or CPU model nor on the number of configured threads. This key point ensures that a compressed file can be decompressed using a different hardware or software configuration.<p>How is this possible? Does it use floating point and concurrency?<p>Cross-platform floating point determinism is seriously difficult. The Rapier physics engine could do it [0] at the expense of disabling simd and multithreading. It also works only on platforms that strictly comply to IEEE 754-2008 which I think that GPUs usually don&#x27;t qualify (regarding subnormal numbers etc). Another thing that may have issues is fused multiply-add which may give higher precision than doing multiplication and addition separately (I think some platforms don&#x27;t have FMA in hardware)<p>For example, it seems that TSAC currently runs on CPUs and nvidia GPUs. Could porting to AMD GPUs affect determinism?<p>[0] <a href="https:&#x2F;&#x2F;rapier.rs&#x2F;docs&#x2F;user_guides&#x2F;rust&#x2F;determinism&#x2F;" rel="nofollow">https:&#x2F;&#x2F;rapier.rs&#x2F;docs&#x2F;user_guides&#x2F;rust&#x2F;determinism&#x2F;</a>

I attempted some ML-as-&#x27;compression&#x27; experiments ~2 years ago, ended up hitting a wall. Check out samples&#x2F;pitch here: <a href="https:&#x2F;&#x2F;lorinhalpert.com&#x2F;ipoc&#x2F;ala&#x2F;" rel="nofollow">https:&#x2F;&#x2F;lorinhalpert.com&#x2F;ipoc&#x2F;ala&#x2F;</a><p>If someone has audio encoding, playback, and&#x2F;or DSPs experience email me to be invited to our our Discord server so we can take another crack at it! :)

So, let me get this straight.<p>Using a ~300 MB model, on a 1 TB hard drive, at 8 Kb&#x2F;s, we can store... ~30 years of music.

This appears to sit just in the middle of something that could be used for music, with a higher bit rate, still much lower than other competing codecs, and something very effective for voice communication, but shrinking the bandwidth (thus the bit rate) also to limit artifacts. Not an expert in the field, anyway I think the supplied examples aren&#x27;t the best ones to show its potential.

Finally, a bit-rate where I can tell the difference between compressed and original!

Next step: use a 1000B LMM (Large Music Model) trained on 1000+ TB of music for zero shot retrieval of any possible sound

Reading this I was wondering how far are video compression (with transformers), turns out decoding is still too expensive in practice (under 10 FPS for 1080p video).<p><a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2206.07307" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2206.07307</a> <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2210.13827" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2210.13827</a>

FYI it looks to be MIT&#x2F;BSD license.<p>Separately:<p>&gt;The Transformer model is evaluated in a deterministic and reproducible way. Hence the result does not depend on the exact GPU or CPU model nor on the number of configured threads.<p>That&#x27;s neat. So even though it&#x27;s &quot;AI-based&quot; its output is guaranteed to be the same for a given input?

I wonder where this codec is on complexity &#x2F; bitrate graph from this post - <a href="https:&#x2F;&#x2F;phoboslab.org&#x2F;log&#x2F;2023&#x2F;02&#x2F;qoa-time-domain-audio-compression" rel="nofollow">https:&#x2F;&#x2F;phoboslab.org&#x2F;log&#x2F;2023&#x2F;02&#x2F;qoa-time-domain-audio-comp...</a>

Pretty good! EnCodec also comes to mind as a neural codec: <a href="https:&#x2F;&#x2F;ai.honu.io&#x2F;papers&#x2F;encodec&#x2F;samples.html" rel="nofollow">https:&#x2F;&#x2F;ai.honu.io&#x2F;papers&#x2F;encodec&#x2F;samples.html</a>

Are there standard-ish codec comparison processes that we can run to see how much perceived fidelity is lost in compression here?

Reminds me of the old IBM &#x27;RECOVC&#x27; codec from around 2000 where they compressed Mel-bank speech.<p><a href="https:&#x2F;&#x2F;ieeexplore.ieee.org&#x2F;document&#x2F;7075313" rel="nofollow">https:&#x2F;&#x2F;ieeexplore.ieee.org&#x2F;document&#x2F;7075313</a><p>All the patents around that are long-dead so good time to do an updated version I guess.<p>If you wanted to do something similar but with way lower bitrates (e.g. 300bps), then look at the NRV codec:<p><a href="https:&#x2F;&#x2F;www.researchgate.net&#x2F;publication&#x2F;224209493_300_bps_noise_robust_vocoder" rel="nofollow">https:&#x2F;&#x2F;www.researchgate.net&#x2F;publication&#x2F;224209493_300_bps_n...</a>

What&#x27;s Nvidia specific about it?

Bellard strikes again...<p>Are we almost converting music to MIDI at this point?<p>As I understand it the model is learning the landscape of sound combinations that are interesting to humans and as such there will be no combination of raw bytes in the recorded file that will result in white noise (for example) being heard because this is never trained for.<p>What if it was though?

How well does it work on any song outside of its training set?