<i>we resize the conditioning (i.e.,
encoded latents) via bilinear sampling to the desired resolution of each stage in the U-Net model</i><p>I am surprised that this works at high compression rates.<p>I would have thought that the more you squeeze into a latent the less correlated the individual latent values become. If they are correlated you can store them more efficiently by having the decoder know about the correlation and store the variance from that correlation with more precision. If that happens, and the values are uncorrelated then bilinear sampling would surely be counterproductive.<p>I feel like even a tiny-brained traditional VAE decoder would be able to do a better job at transforming the latent into a good conditioning for the U-Net.