We replaced Firecracker with QEMU

At CodeSandbox we use Firecracker for hosting development environments, and I agree with the points. Though I don&#x27;t think that means you should not use Firecracker for running long-lived workloads.<p>We reclaim memory with a memory balloon device, for the disk trimming we discard (&amp; compress) the disk, and for i&#x2F;o speed we use io_uring (which we only use for scratch disks, the project disks are network disks).<p>It&#x27;s a tradeoff. It&#x27;s more work and does require custom implementations. For us that made sense, because in return we get a lightweight VMM that we can more easily extend with functionality like memory snapshotting and live VM cloning [1][2].<p>[1]: <a href="https:&#x2F;&#x2F;codesandbox.io&#x2F;blog&#x2F;how-we-clone-a-running-vm-in-2-seconds" rel="nofollow noreferrer">https:&#x2F;&#x2F;codesandbox.io&#x2F;blog&#x2F;how-we-clone-a-running-vm-in-2-s...</a><p>[2]: <a href="https:&#x2F;&#x2F;codesandbox.io&#x2F;blog&#x2F;cloning-microvms-using-userfaultfd" rel="nofollow noreferrer">https:&#x2F;&#x2F;codesandbox.io&#x2F;blog&#x2F;cloning-microvms-using-userfault...</a>

Someone posted this and then immediately deleted their comment: <a href="https:&#x2F;&#x2F;qemu.readthedocs.io&#x2F;en&#x2F;latest&#x2F;system&#x2F;i386&#x2F;microvm.html" rel="nofollow noreferrer">https:&#x2F;&#x2F;qemu.readthedocs.io&#x2F;en&#x2F;latest&#x2F;system&#x2F;i386&#x2F;microvm.ht...</a><p>I didn&#x27;t know it existed until they posted, but QEMU has a Firecracker-inspired target:<p>&gt; <i>microvm is a machine type inspired by Firecracker and constructed after its machine model.</i><p>&gt; <i>It’s a minimalist machine type without PCI nor ACPI support, designed for short-lived guests. microvm also establishes a baseline for benchmarking and optimizing both QEMU and guest operating systems, since it is optimized for both boot time and footprint.</i>

&quot;the fork was very very bad for eating soup - this is a story about how we migrated to a spoon&quot;<p>...firecracker does fine what it was designed to - short running fast start workloads.<p>(oh, and the article starts by slightly misusing a bunch of technical terms, firecracker&#x27;s not technically a hypervisor per se)

<i>&quot;Firecracker&#x27;s RAM footprint starts low, but once a workload inside allocates RAM, Firecracker will never return it to the host system.&quot;</i><p>Firecracker has a balloon device you can inflate (ie: acquire as much memory inside the VM as possible) and then deflate... returning the memory to the host. You can do this while the VM is running.<p><a href="https:&#x2F;&#x2F;github.com&#x2F;firecracker-microvm&#x2F;firecracker&#x2F;blob&#x2F;main&#x2F;docs&#x2F;ballooning.md">https:&#x2F;&#x2F;github.com&#x2F;firecracker-microvm&#x2F;firecracker&#x2F;blob&#x2F;main...</a>

I really want VM&#x27;s to integrate &#x27;smarter&#x27; with the host.<p>For example, if I&#x27;m running 5 VM&#x27;s, there is a good chance that many of the pages are identical. Not only do I want those pages to be deduplicated, but I want them to be zero-copy (ie. not deduplicated after-the-fact by some daemon).<p>To do that, the guest block cache needs to be integrated with the host block-cache, so that whenever some guest application tries to map data from disk, the host notices that another virtual machine has already caused this data to be loaded, so we can just map the same page of already loaded data into the VM that is asking.

No mention of Cloud Hypervisor [1]…perhaps they don’t know about it? It’s based in part on Firecracker and supports free page reporting, virtio-blk-pci, PCI passthrough, and (I believe) discard in virtio-blk.<p>[1]: <a href="https:&#x2F;&#x2F;www.cloudhypervisor.org&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;www.cloudhypervisor.org&#x2F;</a>

The article did an ok job of explaining the firecracker limitations they ran into but it was extremely skimpy when it came to qemu and just rushed to the conclusion “we did a lot of work so try our product.”

Fly uses Firecracker, and they host long-running processes. I wonder what&#x27;s their opinion about it.

&gt; The main issue we&#x27;ve had with QEMU is that it has too many options you need to configure. For instance, enabling your VM to return unused RAM to the host requires at least three challenging tasks<p>This just works on Hyper-V Linux guests btw. For all the crap MS gets they do some things very right.

Presumably this doesn&#x27;t use the &quot;microvm&quot; machine type in QEMU? (also on front page right now <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=36673945">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=36673945</a>)

I came to the same conclusion as OP. QEMU is the most stable, hackable, well-supported VM hypervisor on the market. Setting it up is a pain, but once you get it set up with all your custom scripts, you never have to do it again. Ever. Even in your next project.

I toyed with it a bit and was delighted to get it running. Only to discover getting even basic networking going is another mission in itself.<p>Light is cool but for many tasks that level of Spartan is overkill<p>If I’m investing time in light it might as well be wasm tech

I know that Firecracker does not let you bind mount volumes, but QEMU does. So, we changed to QEMU from Firecracker. If you run the workloads in Kubernetes, you just have to change a single value in a yaml file to change the runtime.<p>I would be scared to let unknown persons use QEMU that bind mounts volumes as that is a huge security risk. Firecracker, I think, was designed from the start to run un-sanitized workloads, hence, no bind mounting.

I know a good way to make a process make the most of the hardware and play cooperatively with other processes: don&#x27;t use virtualization.<p>I will never understand the whole virtual machine and cloud craze. Your operating system is better than any hypervisor at sharing resources efficiently.

Tl;dr: We tried to misuse technology and we failed. If Firecracker was developed for a single binary executed fir a short period of time why do you try to use it for multiple executables running for a long time? Does it make any sense to even try?

Listen people, Firecracker is NOT A HYPERVISOR. A hypervisor runs right on the hardware. KVM is a hypervisor. Firecracker is a process that controls KVM. If you want to call firecracker (and QEMU, when used in conjunction with KVM) a VMM (&quot;virtual machine monitor&quot;) I won&#x27;t complain. But please please please, we need a word for what KVM and Xen are, and &quot;hypervisor&quot; is the best fit. Stop using that word for a user-level process like Firecracker.

<i>&quot;Firecracker...&#x27;s excellent for running short-lived workloads...A little-known fact about Firecracker is its lack of support... for long-lived workloads.&quot;</i><p>Okay.