Stable Diffusion animation

Andreas, author of the Replicate model here -- though &quot;author&quot; feels wrong since I basically just stitched two amazing models together.<p>The thing that really strikes me is that open source ML is starting to behave like open source software. I was able to take a pretrained text-to-image model and combine it with a pretrained video frame interpolation model and the two actually fit together! I didn&#x27;t have to re-train or fine tune or map between incompatible embedding spaces, because these models can generalize to basically any image. I could treat these models as modular building blocks.<p>It just makes your creative mind spin. What if you generate some speech with <a href="https:&#x2F;&#x2F;replicate.com&#x2F;afiaka87&#x2F;tortoise-tts" rel="nofollow">https:&#x2F;&#x2F;replicate.com&#x2F;afiaka87&#x2F;tortoise-tts</a>, generate an image of an alien with Stable Diffusion, and then feed those two into <a href="https:&#x2F;&#x2F;replicate.com&#x2F;wyhsirius&#x2F;lia" rel="nofollow">https:&#x2F;&#x2F;replicate.com&#x2F;wyhsirius&#x2F;lia</a>. Talking alien! Machine learning is starting to become really fun, even if you don&#x27;t know anything about partial derivatives.

Maybe I can shine some light on the debate from an concept artist standpoint that works in VFX and advertising. I worked on feature films (3 of them in the imdb top 100), tv shows (like game of thrones) and hundreds of AD campaigns.<p>In the last 10 year the work of a concept artist changed dramatically we have gone from purely painted concept art to mostly &quot;photobashed&quot;. Photobashed means basically that you rip apart other images and stitch them together to get the desired image. Some start with a rough sketch for the composition or make really rough grey shade 3d model and &quot;overpaint&quot; them. When it comes to &quot;photobashing&quot; the disregard for copyrights was always there and it&#x27;s the worst in smaller studios and a bit better in the leading ones. Still most of the time everyone argues that if you only use really small parts of the images it is covered by fair use. There are some examples were studios got sued but mostly without bigger financial impact.<p>A few months ago I started working with &quot;DiscoDiffusion&quot; to generate the images I use to photobash. &quot;DiscoDiffusion&quot; can produce great &quot;painterly&quot; images but struggles with photorealism and is slower, not as coherent as &quot;StableDiffusion&quot;. Still the adoption rate in the concept art community was insanely fast. This all got topped by &quot;StableDiffusion&quot; in the last week. Ofc there are still people that want to do it the &quot;right&quot; way and not use AI but we had the same discussion years ago when &quot;photobashing&quot; came into place and some artists still wanted to paint the whole image. As concept artist you are mostly paid for your design thinking that means it is less about the process and more about the finished product. The turnaround time for styleframes got reduced from 3-4 hours while painting to 45 min - 1 hour when photobashing with stable diffusion me and my peers in the studio are now at 20-45 min per styleframe. When &quot;photobashing&quot; most people constrain themselves on their image library and ressources like Photobashing Kits. Not only does &quot;StableDiffusion&quot; cut the time in half it also gives greater freedom in composition and design especially if you are using img2img.<p>So where does this leave us? For the work in fast paced art environments like VFX, games, conept art or advertising &quot;StableDiffusion&quot; is a welcome gamechanger. Tradionalists and Artists outside of the industry might feel threatened but for us in these industries it&#x27;s a god send.

I just came across this on twitter, every frame appears to be an evolution of the previous frame using img2img paired with a tilt&#x2F;zoom to create a psychedelic animation.<p>The author claims to have made this with Stable Diffusion, Disco, and Wiggle: <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=Nz_n0qxqoPg" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=Nz_n0qxqoPg</a><p>I believe Wiggle is used to automate the tilt&#x2F;zoom between frames.

I feel like I’m watching an explosion of progress in AI image generation in real-time.<p>Every day there’s a new application of Stable Diffusion. It’s incredible to watch unfold

Very cool! I generated this with 1000 images <a href="https:&#x2F;&#x2F;twitter.com&#x2F;UnshushProject&#x2F;status&#x2F;1563158214577094657" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;UnshushProject&#x2F;status&#x2F;156315821457709465...</a> using the Deforum&#x27;s Colab[1], it&#x27;s really easy and now has interpolation too. It was the very first video, I could have made something great but, you know, awesome guys keep releasing AI tech and I&#x27;m like a child at Luna Park right now, not able to concentrate.<p>If you are interested in my project (I doubt, you are too busy playing like me) I&#x27;m posting a lot of things on <a href="https:&#x2F;&#x2F;unshush.com" rel="nofollow">https:&#x2F;&#x2F;unshush.com</a> and on the Instagram account: <a href="https:&#x2F;&#x2F;www.instagram.com&#x2F;unshushproject&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.instagram.com&#x2F;unshushproject&#x2F;</a> (Sorry for posting my stuff but I&#x27;m not very social so no one will ever see them)<p>If you want to generate videos I can share some links I bookmarked of software&#x2F;code to make them more smooth.<p>[1] Deforum&#x27;s Colab (based on Stable Diffusion): <a href="https:&#x2F;&#x2F;colab.research.google.com&#x2F;github&#x2F;deforum&#x2F;stable-diffusion&#x2F;blob&#x2F;main&#x2F;Deforum_Stable_Diffusion.ipynb" rel="nofollow">https:&#x2F;&#x2F;colab.research.google.com&#x2F;github&#x2F;deforum&#x2F;stable-diff...</a>

AI in animation has been interesting to me for a while now. It leaves me a little conflicted though. If we get to the point where we can throw key drawings at AI and let it handle all the inbewteens without a bunch of tweaking and cleanup afterwards it&#x27;s going to really suck for places like Korea! I guess all those inbeatweeners will just be another victim of automation.<p>I&#x27;ve always loved animation, but I&#x27;ll admit part of that comes from the hubris involved. It&#x27;s pure insanity that people ever drew, by hand, mountains of individual drawings each slightly changed and assembled them into compelling illusions to tell stories. The amount of work that goes into animation is just staggering and anyone sensible would have rejected the entire concept as absurd. I wonder if animation will start losing part of its magic for me when it&#x27;s done primarily by AI.<p>On the other hand though, another thing I&#x27;ve always loved about animation as a storytelling medium is that it isn&#x27;t as limited by practical concerns like physics or reality. If something can be imagined, it can be drawn and animated if somebody has the skill and the resources to fund the massive amounts of work. It&#x27;s time&#x2F;money that forces animators to take shortcuts and make compromises. Creative decisions are made and rejected all the time due to those constraints. If AI driven animation gets more advanced to the point where that&#x27;s no longer such a barrier it could create output more in line with the vision of creators and that&#x27;s exciting too!<p>I hope that traditional hand drawn animation never dies, but I look forward to seeing how AI continues to change the industry and the output.

In this example, 25 frames are generated using Stable Diffusion, then frames are interpolated using FILM-Net. I hadn&#x27;t see FILM-net before, it looks really neat.

It&#x27;s clear that the next frontier is to have 3D-space instead of image space transitions. Language itself is very static and action verbs are not enough to specify scene dynamics. I suppose we would need: A. an enriched version of natural language that refines the dynamic processes that occur in a scene B. a data set of isolated processes labeled in the language described in A.<p>I&#x27;ve had a hard time finding ongoing work on A. and B, perhaps it isn&#x27;t much of a priority for research groups.

There&#x27;s also <a href="https:&#x2F;&#x2F;gist.github.com&#x2F;karpathy&#x2F;00103b0037c5aaea32fe1da1af553355" rel="nofollow">https:&#x2F;&#x2F;gist.github.com&#x2F;karpathy&#x2F;00103b0037c5aaea32fe1da1af5...</a>

Last year, when 3090 GPUs were astronomically priced, I thought &quot;screw it, I&#x27;ll just buy an RTX A5000 for a couple of hundred bucks more.&quot; Which begat a second A5000 for &quot;reasons.&quot; It was almost prescient. Now all these models are coming out requiring slightly higher VRAM GPUs than a 3090, i.e. more in the range of the A5000, and I get to run them. I am a kid in a candy store this past couple of weeks.

I played a bit with smoothly interpolating between Stable Diffusion prompts and the effect can be pretty cool but it&#x27;s hard to avoid discontinuities (like the object changing its orientation), even when using some additional tricks like reusing the previous frame as the initial image or generating several new frames and choosing the one that&#x27;s closest to the previous frame. You basically have to get lucky with the seed. It probably makes most sense to just wait for video models that take temporal consistency into account explicitly or generate 3D models. There is a lot of promising research out there already, so it&#x27;s just a matter of time.

My predictions for 10 - 15 years:<p>- Mandela effect for famous art pieces: &quot;Monalisa was AI generated&quot; &quot;No it wasn&#x27;t&quot; &quot;Yes it was&quot;.<p>- Art critics will get the last laugh, as people start giving them truckloads of money to ask whether a piece of art is human or AI generated.

I&#x27;ve been playing around w&#x2F; StableDiffusion animation using the &quot;deforum&quot; notebook. It&#x27;s taken a bit to really understand how to get results I like with it, but I&#x27;m super happy with how this one came out:<p><a href="https:&#x2F;&#x2F;twitter.com&#x2F;dreamwieber&#x2F;status&#x2F;1565008078466326528?s=21&amp;t=93XvRkPe07uzwpikT4GXaw" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;dreamwieber&#x2F;status&#x2F;1565008078466326528?s...</a><p>It&#x27;s a pretty magical time with this tech. Things are moving very rapidly and I feel excited the way I did when I first rendered 3d animation on my 286 from RadioShack.

Wow, this is incredible. AI tech has been so interesting to follow along lately.<p>Is there something like an index of cool new AI projects that is easy to follow? HN works for this to an extent but I’d love to track more closely.

It is incredible how fast this thing is progressing. Amazing what you can do with some very smart people and 4,000 A100 cluster !<p>What is getting very clear though and this link proves it out is that &#x27;prompt engineering&#x27; is really a thing, I tried this out and it took a while to get something I would consider half decent.<p>I feel like there is a space here for tools &#x2F; technologies to &#x27;suggest&#x27; prompts based on understanding user intentions. If anyone is actually working on this then reach out to me. Email on profile.

Will models like Stable Diffusion be useful for self-driving car research? Like you&#x27;ve got this large NN with weights that are useful for this vision-adjacent task, it should have learned concepts such as edge detection, which could serve as pretrained weights for a self-driving NN?

Amazing! So counting years now when a first AI feature film hits the cinema screens. Source code: screenplay text. I imagine it may look a bit like &quot;A Scanner Darkly&quot; (2006).

Brace yourselves, Cartoon AI Network, coming soon xD

I guess it is missing the physics simulation between frames. Perhaps that is the next big step for ML to get right.

Great, Miyazaki and Lasseter can finally retire.

incredible!

Ok

Com.Samsung.Android.Game.gos:2290:9908:313b42360002

Com.Samsung.Android.Game.gos:2290:9908:313b42360002