Fly.io has GPUs now

Does it have basic functioning other stuff? I am <i>shocked</i> at how our production usage of Fly has gone. Even basic stuff as support not being able to just... look up internal platform issues. Cryptic&#x2F;non-existent error messages. I&#x27;m not impressed. It feels like it&#x27;s compelling to those scared of or ignorant of Kubernetes. I thought I was over Kubernetes, but Fly makes me miss it.

Hi, author of the post and Fly.io devrel here in case anyone has any questions. GPUs went GA yesterday, you can experiment with them to your heart&#x27;s content should the fraud algorithm machine god smile upon you. I&#x27;m mostly surprised my signal post about what the &quot;GPUs&quot; are didn&#x27;t land well here: <a href="https:&#x2F;&#x2F;fly.io&#x2F;blog&#x2F;what-are-these-gpus-really&#x2F;">https:&#x2F;&#x2F;fly.io&#x2F;blog&#x2F;what-are-these-gpus-really&#x2F;</a><p>If anyone has any questions, fire away!

As far as I know, Fly uses Firecracker for their VMs. I&#x27;ve been following Firecracker for a while now (even using it in a project), and they don&#x27;t support GPUs out of the box (and have no plan to support it [1]).<p>I&#x27;m curious to know how Fly figured their own GPU support with Firecracker. In the past they had some very detailed technical posts on how they achieved certain things, so I&#x27;m hoping we&#x27;ll see one on their GPU support in the future!<p>[1]: <a href="https:&#x2F;&#x2F;github.com&#x2F;firecracker-microvm&#x2F;firecracker&#x2F;issues&#x2F;1179#issuecomment-1846090947">https:&#x2F;&#x2F;github.com&#x2F;firecracker-microvm&#x2F;firecracker&#x2F;issues&#x2F;11...</a>

It’s cool to see that they can handle scaling down to zero. Especially for working on experimental sites that don’t have the users to justify even modest server costs.<p>I would love an example on how much time a request charges. Obviously it will vary, but is it 2 seconds or “minimum 60 seconds per spin up”?

I was an early adopter of Fly.io. It is not production-ready. They should fix their basic features before adding new ones.

About Fly but not about the GPU announcement, I wish they had a S3 replacement, they suggest a GNU Affero project that is a dealbreaker for any business, needing to leave Fly to store user assets was a dealbreaker for us to use Fly on our next project, sad cause I love the simplicity, the value for money, the built in VPN.

Who is the target market for this? Small&#x2F;unproven apps that need to run some AI model, but won&#x27;t&#x2F;can&#x27;t use hosted offerings by the literally dozens of race-to-zero startups offering OSS models?<p>We run plenty of our own models and hardware, so I get wanting to have control over the metal. I&#x27;m just trying to figure out who <i>this</i> is targeted at.

The recipe example or any any LLM use case seems like a very poor way of highlighting “inference at the edge” given the extra few hundred ms round trip won’t matter.

Anybody has experience with the performance. First glance is that they are quite expensive. Compared to for example Hetzner (cpu machines)

I use Fly.io free tier to run uptime monitoring with Uptime kuma. It works insanely well, and I&#x27;m a really happy camper.

I don&#x27;t want to deploy an app, I just want to play around with LLMs and don&#x27;t want to go out and buy an expensive PC with a highend GPU just now. Is Fly.io a good way to go? What about alternatives?

Has anyone who has used Beam.Cloud compare that service to this one?

Great, more competition for the price-gouging platforms like Replicate and Modal is needed. As always with these, I would be curious about the cold-start time -- are you doing anything smart about being able to start (load models into VRAM) quickly? Most platforms that I tested are completely naive in their implementation, often downloading the docker image just-in-time instead of having it ready to be deployed on multiple machines.

It is a bit of an odd thing that we still call GPUs GPUs when the main use for them seems to have little to do with Graphics!

How fast is the spin up&#x2F;down on this scale to zero? If it is fast this could be pretty interesting

Interesting. We have this discussing this kind of services (offloading training) over the last several days [1] [2] [3]. Thinking on the opportunity to compete with top cloud services such as Google Cloud, AWS, and Azure.<p>[1] <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=39353663">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=39353663</a><p>[2] <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=39329764">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=39329764</a><p>[3] <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=39263422">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=39263422</a>

Is there any configuration to keep alive the machine for X seconds after a request has been served, instead of scaling down to zero immediately? I couldn&#x27;t find it skimming the docs

Can fly run cog files like replicate uses? Would be nice to take those pre packaged models run them here with the same prediction API<p>Maybe cos it&#x27;s replicate they might be hesitant to adopt it but it does seem to make things a lot smoother Even with lambalabs&#x27; lambdastack I still hit cuda hell <a href="https:&#x2F;&#x2F;github.com&#x2F;replicate&#x2F;cog">https:&#x2F;&#x2F;github.com&#x2F;replicate&#x2F;cog</a>

Almost twice as cheap as Modal! Very nice!

Somehow cheaper than AWS?

anyone know if this is a PCI passthrough for a full a100? or some fancy clever vgpu thing?

This is amazing and it shows that Nvidia should be the most valuable stock in the world. Every company, country, city, town, village, large enterprise, medium and small business, AI bro, Crypto bro, gamer bro, big tech, small tech, old tech, new tech, and start up want Nvidia GPUs. Nvidia GPUs will become the new green oil of the 21st century. I am all in and nothing short of a margin call will change my mind.

too expensive

Now having GPUs is news now?

&gt; The speed of light is only so fast<p>This is the title of one of the sections. Why? Think IT sector needs to stop using such titles.