Pillow-SIMD – Fast, production-ready image resize for x86

Is doing lanczos really faster and&#x2F;or better quality for scaling down than just doing a simple pixel mixing:<p><a href="http:&#x2F;&#x2F;entropymine.com&#x2F;imageworsener&#x2F;pixelmixing&#x2F;" rel="nofollow">http:&#x2F;&#x2F;entropymine.com&#x2F;imageworsener&#x2F;pixelmixing&#x2F;</a><p>I implemented this for my image pipeline:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;pedrocr&#x2F;rawloader&#x2F;blob&#x2F;230432a403a9febb5e5c004780211f7e765130b9&#x2F;src&#x2F;imageops&#x2F;demosaic.rs#L155-L190" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;pedrocr&#x2F;rawloader&#x2F;blob&#x2F;230432a403a9febb5e...</a><p>Makes for simple enough code and even before any serious effort at optimization or SIMD it can convert a 3680x2456x4 image in 32bit float (source article is 3x8bit) to 320x200x4 also in 32bit float in about 60ms (across 4 threads in 2 cores on a i5-6200U).

Let me insert my personal pet-peeve: have you thought of making it colour-space-aware? Most (all?) images you&#x27;ll encounter are stored in sRGB colourspace, which isn&#x27;t linear, so you can&#x27;t do the convolution by just multiplying and adding (the result will be slightly off). The easiest way would be to convert it to 16-bit-per-channel linear colour space using a lookup table, do the convolution in linear 16-bit space, then convert back to 8-bit sRGB.

I&#x27;d be very interested in an optional Pillow-SIMD downsampling resize that produces 16 bit output internally and then uses a dither to convert from 16 bit to 8 bit. Photoshop does this by default and it produces superior downsampling. Without keeping the color resolution higher, you can end up with visible color banding in resized 8 bit images that wasn&#x27;t visible in the source image.<p>I am curious if the reason that Pillow-SIMD is more than 4x faster than IPP is due to features IPP supports - like higher internal resolution - that Pillow-SIMD doesn&#x27;t? The reported speeds here are amazing, and I&#x27;m definitely going to check this project out and probably use it, but I&#x27;d love a little clarity on what the tradeoffs are against IPP or others. I assume there are some.

Obligatory: <a href="http:&#x2F;&#x2F;johncostella.webs.com&#x2F;magic&#x2F;" rel="nofollow">http:&#x2F;&#x2F;johncostella.webs.com&#x2F;magic&#x2F;</a>

Unrelated: since when has this picture of Bologna become a new lenna.jpg?<p>I think I have already seen it in a couple of recent posts about image compression. (Fits perfectly the definition of Baader-Meinhof phenomenon [1].)<p>[1] <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;List_of_cognitive_biases#Frequency_illusion" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;List_of_cognitive_biases#Frequ...</a>

FWIW the Accelerate framework[1] gives roughly comparable performance[2] for Lanczos resizing. Apple platforms only, but all Apple platforms, not limited to x86.<p>[1] vImageScale_ARGB8888( ).<p>[2] I don&#x27;t have identical hardware available to time on, and it&#x27;s doing an alpha channel as well, so this is slightly hand-wavy.

Curious how this would compare vs. running it on the GPU? This is literally what GPUs are made for, and they often have levels of parallelism 500+ times greater than SIMD.

I&#x27;m really happy to see this. The one time I tried looking at the PIL sources for resizing, I was appalled at what I saw. Simply seeing that you&#x27;re expanding the filter size as the input to output ratio shrinks is a huge deal.<p>When I wrote my own resizing code, I found it helpful to debug using a nearest-neighbor kernel: 1 from -0.5 to 0.5 and 0 everywhere else. It shook out some off-by-one errors.

&gt; No tricks like decoding a smaller image from a JPEG<p>Given that most cameras are producing JPEG now, I&#x27;m curious why you don&#x27;t make use of the compressed &#x2F; frequency-domain representation. To a novice in this area (read: me), It seems like a quick shortcut to an 8x or 4x or 2x downsample.<p>Or is the required iDCT operation just that much more expensive than the convolution approach?

Looks really nice!<p>I&#x27;d love to see vips in the benchmark comparison, perhaps a Halide-based resizer too as those are the fastest I&#x27;ve found so far. Perhaps GraphicsMagick too, as I believe it&#x27;s meant to be faster than ImageMagick in many cases.

Another data point: MATLAB, glnxa64 AVX2, 12 core<p>&gt;&gt; maxNumCompThreads(1);<p>&gt;&gt; im = randi(255, [2560, 1600, 3],&#x27;uint8&#x27;);<p>&gt;&gt; timeit(@()imresize(im,[320,200],&#x27;bilinear&#x27;,&#x27;Antialiasing&#x27;,false))<p>ans =<p><pre><code> 0.0083 </code></pre> &gt;&gt; timeit(@()imresize(im,[320,200],&#x27;bilinear&#x27;))<p>ans =<p><pre><code> 0.0301 </code></pre> &gt;&gt; maxNumCompThreads(6);<p>&gt;&gt; timeit(@()imresize(im,[320,200],&#x27;bilinear&#x27;,&#x27;Antialiasing&#x27;,false))<p>ans =<p><pre><code> 0.0062 </code></pre> &gt;&gt; timeit(@()imresize(im,[320,200],&#x27;bilinear&#x27;))<p>ans =<p><pre><code> 0.0113 </code></pre> Oh, missed that lanczos2 part:<p>&gt;&gt; maxNumCompThreads(1);<p>&gt;&gt; timeit(@()imresize(im,[320,200],&#x27;lanczos2&#x27;,&#x27;Antialiasing&#x27;,false))<p>ans =<p><pre><code> 0.0146 </code></pre> &gt;&gt; maxNumCompThreads(6);<p>&gt;&gt; timeit(@()imresize(im,[320,200],&#x27;lanczos2&#x27;,&#x27;Antialiasing&#x27;,false))<p>ans =<p><pre><code> 0.0049 </code></pre> Since MATLAB tries to do most of the computation in double precision, its harder to extract much from SIMD.

Have you tried to use a fast blur (like StackBlur for example : <a href="http:&#x2F;&#x2F;www.quasimondo.com&#x2F;BoxBlurForCanvas&#x2F;FastBlur2Demo.html" rel="nofollow">http:&#x2F;&#x2F;www.quasimondo.com&#x2F;BoxBlurForCanvas&#x2F;FastBlur2Demo.htm...</a> , the radius should be computed according to the ratio between original size and target size) as a first step before taking the classic nearest neighbor ? And also try to make an algorithm that resize to multiple resolution at the same time could improve speed<p><pre><code> I take an image of 2560x1600 pixels in size and resize it to the following resolutions: 320x200, 2048x1280, and 5478x3424 </code></pre> So you are also upscaling ?

Great work and great article. One question though: Did you consider to replace convolution based filters by FFT based ones ?

&gt; I wasn’t building it for fun: I work for Uploadcare and resizing images has always been a practical issue with on-the-fly image processing.<p>you ever consider pushing the work entirely to the client with a resize implemented in javascript? that would cut down on bandwidth as well.

Any reason this had to be a fork?<p>I would much rather this feature be in Pillow so ALL of the python ecosystem could get 6 times faster image resizing.

Looks fantastic! Are there other bottlenecks, such as JPEG encoding and decoding that can be ported to SIMD code in Pillow?

Can the speed increased even further using GPGPU?

&gt; With optimizations, Uploadcare now needs six times fewer servers to handle its load than before.<p>This is devil&#x27;s advocate, but did you guys have concrete need for this optimization? You now need six times fewer servers, but was that a crippling problem, or is it a cool statistic for the future when you get more users?