StreamDiffusion: A pipeline-level solution for real-time interactive generation

Arxiv paper here <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2312.12491" rel="nofollow noreferrer">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2312.12491</a><p>I think that it&#x27;s possible to get faster than their default timings for a 4090 (I have been able to get 10fps without optimizations with SDXL Turbo and 1 iteration step), but their other improvements like using a Stochastic Similarity Filter to prevent unnecessary generations are good for getting fast results w&#x2F;out having to pin your GPU at 100% all the time.

This feels unreal. It feels like a decade passed within a year.

I just tried the realtime-text2img demo (uses npm for frontend which i think is too much for this). Modified it to produce only 1 image instead of 16. Works well on a laptop with RTX-3080. It&#x27;s probably 2 images &#x2F; sec.<p>EDIT: The `examples\screen` demo almost feels realtime. Says 4 fps on the window but don&#x27;t what it represents.<p>EDIT: Denoising in img2img is very low though which means thee returned image is only slightly different from base image.

Does 100fps mean I can provide a new input every 10 ms and get a new output every 10ms? Or do inputs need to be batched together to get that average throughput?

This more or less just worked as documented. Most of these demos tend to blow up and give really wonky deep errors.<p>Good job. Give it a try. Look into the server.py of realtime-txt2img to change the model if you want to generate something other than anime. Pointing it to say <a href="https:&#x2F;&#x2F;huggingface.co&#x2F;runwayml&#x2F;stable-diffusion-v1-5" rel="nofollow noreferrer">https:&#x2F;&#x2F;huggingface.co&#x2F;runwayml&#x2F;stable-diffusion-v1-5</a> works fine.<p>The results are genuinely fast. Not great, but fast. If you change to the SDXL via LCM-LoRA <a href="https:&#x2F;&#x2F;huggingface.co&#x2F;latent-consistency" rel="nofollow noreferrer">https:&#x2F;&#x2F;huggingface.co&#x2F;latent-consistency</a> you may get better stuff but that&#x27;s when it&#x27;s going to get difficult and you&#x27;ll start to run into those mysterious crashes I talked about that require, you know, actual work.<p>my setup: 4090&#x2F;3990x&#x2F;CUDA 12.2&#x2F;debian sid. ymmv.

How does the demo with the girl moving in and out of frame work? Is it ControlNet?

Maybe we&#x27;re all living in a simulation^H^H^H^H^H pipeline-level solution for real-time interactive generation.

What is the fps on Apple Silicon?

Is there a video of it I can view anywhere?

Yo I just heard about MidJourney this year.<p>And this appears to be a local runtime stable diffusion streaming library?<p>Bruh.

This paper is horribly written. It&#x27;s like the authors are trying to sell me on them as researchers, instead of helping me understand their research (y&#x27;know, the entire reason journals got started??). An entire section for &quot;stream batching&quot; was just too much, and none of their ideas were innovative or unique. It was incredibly dense, simply because it&#x27;s obfuscated, which makes me believe the authors themselves don&#x27;t really understand what they&#x27;re doing.<p>The results aren&#x27;t even very good. They claim 60x speedup, but compared to what? HuggingFace&#x27;s Diffusers Autopipeline... a company notorious for buggy code and inefficient pipelines. And that&#x27;s for <i>naively</i> running the pipeline on every image. Give me a break.