My VM is lighter (and safer) than your container (2017)

I&#x27;m quite interested in seeing where slim VM&#x27;s go. Personally I don&#x27;t use Kubernetes, it just doesn&#x27;t fit my client work which is nearly all single-server and it makes more sense to just run podman systemd units or docker-compose setups.<p>So from that perspective, when I&#x27;ve peeked at firecracker, kata containers, etc, the &quot;small dev dx&quot; isn&#x27;t quite there yet, or maybe never will get there since the players target other spaces (aws, fly.io, etc). Stuff like a way to share volumes isn&#x27;t supported, etc. Personally I find Dockers architecture a bit distasteful and Podmans tooling isn&#x27;t <i>quite</i> there yet (but very close).<p>Honestly I don&#x27;t really care about containers vs VMs except the VM alleges better security which is nice, and I guess I like poking at things but they&#x27;re were a little too rough for weekend poking.<p>Is anyone doing &quot;small scale&quot; lightweight vm deployments - maybe just in your homelab or toy projects? Have you found the experience better than containers?

Title is kinda clickbaity (wha-? how can a VM be lighter than a container). It&#x27;s about unikernels.

This reminds me: in 2015 I went to Dockercon and one booth that was fun was VMWare&#x27;s. Basically they had implemented the Docker APIs on top of VMWare so that they could build and deploy VMs using Dockerfiles, etc.<p>I&#x27;ve casually searched for it in the past and it seems to not exist anymore. For me, one of the best parts of Docker is building a docker-image (and sharing how it was done via git). It would be cool to be able to take the same Dockerfiles and pivot them to VMs easily.

This paper references consistently mischaracterizes AWS Lambda as a &quot;Container as a Service&quot; technology, when in fact it is exactly the sort of lightweight VM that they are describing - <a href="https:&#x2F;&#x2F;aws.amazon.com&#x2F;blogs&#x2F;aws&#x2F;firecracker-lightweight-virtualization-for-serverless-computing&#x2F;" rel="nofollow">https:&#x2F;&#x2F;aws.amazon.com&#x2F;blogs&#x2F;aws&#x2F;firecracker-lightweight-vir...</a>

The issue with unikernels and things like Firecracker are that you can&#x27;t run them on already-virtualized platforms<p>I researched Firecracker when I was looking for an alternative to Docker for deploying FaaS functions on an OpenFaaS-like clone I was building<p>It would have worked great if the target deployment was bare metal but if you&#x27;re asking a user to deploy on IE an EC2 or Fargate or whatnot, you can&#x27;t use these things so all points are moot<p>This is relevant if you&#x27;re self-hosting or you ARE a service provider I guess.<p>(Yes, I know about Firecracker-in-Docker, but I mean real production use)

Containers should really be viewed as an extension of packages (like RPM) with a bit of extra sauce with the layered filesystem, a chroot&#x2F;jail and cgroups for some isolation between different software running on the same server.<p>Back in 2003 or so we tried doing this with microservices that didn&#x27;t need an entire server with multiple different software teams running apps on the same physical image to try to avoid giving entire servers to teams that would be only using a few percent of the metal. This failed pretty quickly as software bugs would blow up the whole image and different software teams got really grounchy at each other. With containerization the chroot means that the software carries along all its own deps and the underlying server&#x2F;metal image can be managed seperately, and the cgroups means that software groups are less likely to stomp on each other due to bugs.<p>This isn&#x27;t a cloud model of course, it was all on-prem. I don&#x27;t know how kubernetes works in the cloud where you can conceivably be running containers on metal sharing with other customers. I would tend to assume that under the covers those cloud vendors are using Containers on VMs on Metal to provide better security guarantees than just containers can offer.<p>Containers really shouldn&#x27;t be viewed as competing with VMs in a strict XOR sense.

&gt; We achieve lightweight VMs by using unikernels<p>When I attended Infiltrate a few years ago, there was a talk about unikernels. The speaker showed off how incredibly insecure many of them were, not even offering support for basic modern security features like DEP and ALSR.<p>Have they changed? Or did the speaker likely just cherry-pick some especially bad ones?

What I see happening now on the cloud is containers from different companies and different security domain running on the same VM. I have to think this is fundamentally insecure and that VMs are underrated.<p>I hear people advocate QubesOS for security which is based on XEN when it comes to running my client. They say my banking should be done in a different VM than my email for instance. Well if that’s the case, why do we run many containers doing different security sensitive functions on the same VM when containers are not really considered a very good security boundary?<p>From a security design perspective I imagine hardware being exclusive to a person&#x2F;organization, vms being exclusive to some security function, and containers existing on top of that makes more sense from a security function but we seem to be playing things more loosely on the server side.

Sometimes the less strict separation is a feature, not a bug.<p>Without folder sharing with dockers for example, it would be pretty useless.

It&#x27;s not clear to me that VMs actually do offer better isolation than well-designed containers (i.e. not docker).<p>It&#x27;s basically a question of: do you trust the safety of kernel-mode drivers (for e.g. PV network devices or emulated hardware) for VMs, or do you trust the safety of userland APIs + the limited set of kernel APIs available to containers.<p>On my FreeBSD server, I kind of trust jails with strict device rules (i.e. there are only like 5 things in &#x2F;dev&#x2F;) over a VM with virtualized graphics, networking, etc.

This is 5 years old. What&#x27;s the current state of the art?

[2017]

How does LightVM compare to Firecracker VMs? Could it be used for on-demand cloud VMs?

Not surprising that VMs running unikernels are as nimble as containers, but not quite useful either, at least in general. Much easier to just use a stock docker image.

Kubernetes says no...<p>The article is light on detail. Containers and VMs have different use cases. If you self host lightweight VMs is likely the better path, however, once you in the cloud most managed services only provide support for containers.

Why is a 5yr old article being posted now? If this were to catch on, it would&#x27;ve. I just dont see it being used anywhere.<p>Having a full Linux kernel available is a major benefit that you lose, right?

Yes, when you custom engineer a specific, complex solution for a specific use case, it is generally more performative than a general-use solution that&#x27;s simple.

(2017)

Tell me you don&#x27;t understand containers wihout telling me you dont understand containers.

Containers and VMs are totally not the same thing. They serve a complete other purpose , as multiple containers can be combined to create an application&#x2F;service , VMs always use a complete os etc etc anyway the internet is full of the true purpose of containers , they were never meant to use as a &quot;VM&quot; and about security.. meh everything is insecure until proven differently

&quot;orders of magnitude&quot; :<p>Why does anyone ever write &quot;two orders of magnitude&quot; when 100x is shorter?<p>Of course, this presumes 10 as the magnitude and the N orders to be the exponent, but I don&#x27;t think I&#x27;ve ever, since the 90s, seen that stilted phrasing ever used for a base other than 10.