3D-to-photo is an open source tool for Generative AI product photography, that uses 3D models to allow fine camera angle control in generated images.<p>If you have 3D models created using the iOS 3D scanner you can upload them directly on to 3D-to-photo and describe the scene you want to create. For example:<p>"on a city side walk"
"near a lake, overlooking the water"<p>Then click "generate" to get the final images.<p>The tech stack behind 3D-to-photo:<p>Handling 3d models on the web: @threejs
Hosting the diffusion model: @replicate
3D scanning apps: shopify,Polycam3D or LumaLabsAI
Given that Stable Diffusion is designed to be able to run on consumer hardware, without the need for a third party cloud platform, it saddens me to see that this, alongside many other similar projects, require the use of a third party platform for hosting the model, even for local usage. The tool itself does seem interesting though.
How is this different than just photoshopping a 2d image of a 3d object onto an SD generated background? Is it just meant to let people skip the step of generating a background and compositing? (Sorry, inpainting. But the distinction seems minimal here as people have been photoshopping 3d objects believably into scenes for decades before SD came around)
God we're so close from being able to feed a photo and some measurements into a program and get an accurate model out of it. I can't wait until my smartphone and a set of calipers can replace a $700 3d scanner.
Neat. I recently was having fun with doing something similar manually in Blender, generating depth map and using it in Stable Diffusion with controlnet. Results were great. My models didn't have texture though, so sd generated it. But I imagine I could go with img2img to preserve texture if I had it.
Looks pretty cool. Can anyone comment on how to hack together the opposite? That is, going from 2D object image to 3D rendering with in-painted background? Or is that not possible right now.
I took a quick look at the python flask code and I’m still not sure if there’s a reason of not just using Next’s server side aspects. JS can do every operation I skimmed by<p>Thoughts?