Btw, I did this in pixel space for simplicity, cool animations, and compute costs. Would be really interesting to do this as an LDM (though of course you can't really do the LAB color space thing, unless you maybe train an AE specifically for that color space. )<p>I was really interested in how color was represented in latent space and ran some experiments with VQGAN clip. You can actually do a (not great) colorization of an image by encoding it w/ VQGAN, and using a prompt like "a colorful image of a woman".<p>Would be fun to experiment with if anyone wants to try, would love to see any results if someone wants to build
Author's writeup on this from May: <a href="https://medium.com/@erwannmillon/color-diffusion-colorizing-black-and-white-images-with-diffusion-models-269828f71c81" rel="nofollow noreferrer">https://medium.com/@erwannmillon/color-diffusion-colorizing-...</a>
I’m not a fan of b&w colorization. Often the colors are wrong, either outright color errors (like choices for clothing or cars) or often not taking in to account lighting conditions (late in day shadows but midday brightness).<p>Then there is the issue of B&W movies. Using this kind of tech might not give pleasing results as the colors used for sets and outfits were chosen to work well for film contrast and not for story accuracy. That “blue” dress might really be green. (Please, just leave B&W movies the way they are.)
Does it work on arbitrary image sizes?<p>One of the nice features of the somewhat old Deoldify colorizer is support for any resolution. It actually does better than photoshops colorization: <a href="https://blog.maxg.io/colorizing-infrared-images-with-photoshop/" rel="nofollow noreferrer">https://blog.maxg.io/colorizing-infrared-images-with-photosh...</a><p>Edit - technically, I suppose, the way Deoldify works is by rendering the color at a low resolution and then applying the filter to a higher resolution using OpenCV. I think the same sub-sampling approach could work here...
Is there anything that exists right now with diffusion models to improve poor VHS coloring? The coloring does exist so I would not want to replace a red shirt by a blue shirt for example but it's just not very accurate.
This is a cool party trick, but I don't see a need for this in any real applications. Black and white is its own art form, and a lot of really great black and white images would look like absolute garbage if you could convert them to color. This is because the things that make a great black and white image (dramatic contrasts, emphasis on shape/geometry, texture, etc) can lose a lot of their impact when you introduce color. Our aesthetic tolerance for contrast seems significantly reduced in color because our expectations for the image are more anchored in how things look in the real world. And colors which can be very pleasing in some images are just distracting in others.<p>So all this is to say.... I don't think there would be commercial demand to, say, "upgrade" classic movies with color. Those films were shot by cinematographers who were steeped in the black & white medium and made lighting and compositional choices that take greatest advantage of those creative limitations.
Colourising old photographs is the banal apotheosis application of diffusion AI.<p>It's the pinnacle of the whole thing: "imagine it for me in a way that conforms to my contemporary expectations".<p>If you're going to colourise images, have the decency to do it by hand. If possible on a print with brushes.<p>Edit: didn't think this would be popular. Maybe it's the historical photography nerd in me, but colourising images without effort and thought is like smashing vintage glass windows for the fun of it: cultural vandalism.