Real-Time Adaptive Image Compression

Nice work, but disingenuous to not include a BPG (HEVC) image for comparison -- BPG is close to state-of-the-art, not WebP -- even their own SSIM charts show this.<p>Interesting that decoding is slower than encoding. Also curious about performance on CPU.<p>This approach may also be susceptible to &quot;hallucinating&quot; inaccurate detail; you can see a little bit of this on the upper-right of the girl&#x27;s circled eyelid compared to the original Kodak image. See also: <a href="http:&#x2F;&#x2F;www.dkriesel.com&#x2F;en&#x2F;blog&#x2F;2013&#x2F;0802_xerox-workcentres_are_switching_written_numbers_when_scanning" rel="nofollow">http:&#x2F;&#x2F;www.dkriesel.com&#x2F;en&#x2F;blog&#x2F;2013&#x2F;0802_xerox-workcentres_...</a>

As someone interested in the field, this does not look much different from state-of-the-art video codecs, e.g. variable-size blocks, wavelet, predicator, arithmetic coding. Only that the predicator is trained on real data. But symbol dictionaries are already used in modern compressors like brotli and zstd.<p>Most codecs have been tuned for a mean subjective score (MOS). MS-SSIM is not particularly good to fully rely on [1] [2]. In my experiments it performed poorly.<p>I think Google team effort will have a much bigger impact [3] just by combining all recent practical improvements in the image compression.<p>Meanwhile, they could have optimized images on the website a little better. Saved ~15% with my soon-to-be-obsolete tool [4].<p>[1] <a href="https:&#x2F;&#x2F;encode.ru&#x2F;threads&#x2F;2738-jpegraw?p=52583&amp;viewfull=1#post52583" rel="nofollow">https:&#x2F;&#x2F;encode.ru&#x2F;threads&#x2F;2738-jpegraw?p=52583&amp;viewfull=1#po...</a><p>[2] <a href="https:&#x2F;&#x2F;medium.com&#x2F;netflix-techblog&#x2F;toward-a-practical-perceptual-video-quality-metric-653f208b9652" rel="nofollow">https:&#x2F;&#x2F;medium.com&#x2F;netflix-techblog&#x2F;toward-a-practical-perce...</a><p>[3] <a href="https:&#x2F;&#x2F;encode.ru&#x2F;threads&#x2F;2628-Guetzli-a-new-more-psychovisual-JPEG-encoder?p=52198&amp;viewfull=1#post52198" rel="nofollow">https:&#x2F;&#x2F;encode.ru&#x2F;threads&#x2F;2628-Guetzli-a-new-more-psychovisu...</a><p>[4] <a href="http:&#x2F;&#x2F;getoptimage.com" rel="nofollow">http:&#x2F;&#x2F;getoptimage.com</a>

From their website:<p>&gt; Lubomir holds a Ph.D. from UC Berkeley, 20 years of professional experience, 50+ issued patents and 5000+ citations.<p>I just hope this type of research isn&#x27;t going to end in a patent encumbrance, like it did with JPEG and MPEG.<p>These techniques are right around the corner, no matter who invents the file formats.<p>So if their idea is to lock these general ideas down with more patents, I&#x27;d want them to stop their research and let people with more open intentions research this further.

A rarely discussed danger of all machine learning models: If they don&#x27;t know the answer, they&#x27;ll rather make something up.<p>Here&#x27;s a Google Translate example: <a href="https:&#x2F;&#x2F;twitter.com&#x2F;keff85&#x2F;status&#x2F;862690920805916672" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;keff85&#x2F;status&#x2F;862690920805916672</a><p>I wouldn&#x27;t like to lose a part of parcel in a lawsuit because an adaptive algorithm made up some details in aerial photograph so that it compresses better ...

So is this only good for ridiculously low target filesizes? Noone in their right mind is going to compress a &quot;480x480 image to a file size of 2.3kB&quot;<p>What I want to see is an acceptable looking JPG next to a WaveOne image of the same size. Or an acceptable looking WaveOne next to a JPG of the same size.<p>How small is good enough? How good is small enough?

I wonder how this compares to FLIF. I also tried to compress images based on shape and structure but by approximating these using skeletons.<p>I&#x27;m just a bit struggling with their performance comparison. The graphs they present are very pretty and promising but for the presented images we&#x27;re quite left in the dark. They dump some images and theirs looks prettier and the authors give us <i>some</i> indication of quality but it&#x27;s not conclusive evidence that their method produces better images. Typically when different compressed images are presented two things can vary: quality and file-size. In the presented images both seem to varying without telling us which is which. Also, there is no baseline to compare against, either in terms of filesize or what the should look like. Sure, we humans can make a very educated guess but it is just sloppy to not include the uncompressed original image.<p>I will be fully convinced when I can try it for myself on my own image set.

Very impressive work, though it seems like a mistake to focus on compression, which gets less valuable as storage and bandwidth gets cheaper. You need only look to the staying power of jpeg, which is so far from the state of the art, yet it&#x27;s not going anywhere. Why? The demand for replacing it is not strong enough.<p>They obviously have some good image priors here, if I were them I would consider applying this tech to other image-related things, like image manipulation, or image search. Although competition is heating up quickly in these fields...

Deep learning will be a great way to do compression for sure, both for audio, video and images. I could see that one could download &quot;knowledge sets&quot; for these decompressors. Looking at Google Earth, download the supplemental &quot;knowledge set&quot; for overhead shots of cities and country side. Looking at people, download the supplemental &quot;knowledge set&quot; for faces and clothing, etc.<p>Basically each domain you want to do well in you need a knowledge set that is trained on that data. Then you need a discriminator on the compression side to classify an image or subregions of an image into those categories.<p>If you can make the knowledge sets downloadable on demand and then cached you can be incredibly efficient over the long term, while maintaining very small initial download sets. I think evolveable knowledge sets over time also ensure that the codex is flexible to handle currently not foreseen situations. Nobody wants a future where are DL-based image&#x2F;video compression tool only knows a few pre-determined sets and is mediocre on everything else.

This encoder seems to have some weird distortions that are most visible in the aerial shots. Compare [1] and [2]. The lines on the basketball court are distorted and curved. If you look closely, there is also curvature added to the sidewalks where there isn&#x27;t any. In case those links break, I&#x27;m referring to the top row of aerial images.<p>[1] <a href="https:&#x2F;&#x2F;static1.squarespace.com&#x2F;static&#x2F;57c8be4459cc68c3e3d7b040&#x2F;590bade05016e1ca29d9d74f&#x2F;5910f88203596eba4ac0e444&#x2F;1494284425679&#x2F;bloomington28_3_crop_aerial_bpp0.1_disc_reconst_WO.png" rel="nofollow">https:&#x2F;&#x2F;static1.squarespace.com&#x2F;static&#x2F;57c8be4459cc68c3e3d7b...</a><p>[2] <a href="https:&#x2F;&#x2F;static1.squarespace.com&#x2F;static&#x2F;57c8be4459cc68c3e3d7b040&#x2F;590bade05016e1ca29d9d74f&#x2F;5910f88086e6c0368e048208&#x2F;1494284423491&#x2F;bloomington28_3_crop_aerial_bpp0.1_disc_reconst_JP2.png" rel="nofollow">https:&#x2F;&#x2F;static1.squarespace.com&#x2F;static&#x2F;57c8be4459cc68c3e3d7b...</a>

&gt; <i>While we are slightly faster than JPEG (libjpeg) and significantly faster than JPEG 2000, WebP and BPG, our codec runs on a GPU and traditional codecs do not — so we do not show this comparison.</i><p>This is great news!<p>I&#x27;d actually like to see the plot, though. (Both for encoding and decoding.) It stands to reason that a neural network can optimize image compression, as it can encode high-level information like &quot;this is a face&quot;. But encoding &#x2F; decoding speed is the sticking point, so I feel successes there should be emphasized.<p>The necessity of having a GPU doesn&#x27;t seem problematic nowadays; everything has one. Testing it with a mobile-grade GPU would be interesting.

I&#x27;m going to need to see this code in practise to believe it.

Seems like a slightly unfair comparison. Training the compressor moves data from the images into the compressor, making the bit per pixel evaluation slightly more iffy.

One big reason for &quot;hardcoded&quot; encoders and decoders is that they much easier to implement in hardware.<p>One can improve e.g. H.265 somewhat easily, if software only solution is an option. But if you need cheap hardware only solution then ML-required-way seems a bit too expensive.

Does no one realize this is a joke &#x2F; marketing?<p>Directly from the paper&#x27;s PDF:<p>&quot;Finally, Pied Piper has recently claimed to employ ML techniques in its Middle-Out algorithm (Judge et al., 2016), although their nature is shrouded in mystery.&quot;

Is this open source or something that you are aiming to license?

Where is PNG?

for some reaso they do not show the uncompressed image for comparison