Introduction to Immutable Linux Systems

While I am glad Silverblue is on this list, not having Fedora CoreOS on it too is a shame. FCOS is an amazing OS to run in production and it has come a very long way since the CoreOS acquisition. I find that FCOS is a good middle ground of being usable and easy to learn while still being immutable compared to Nix.<p>The FCOS devs introduced a new feature called CoreOS Layering which lets you define your system in a Dockerfile and FCOS will rebase to that state and all you have to do is reboot to configure your server. It is super powerful.<p>Anyways, your next project needs a VM, give it a shot. I made a Python based CLI tool to help you develop locally on a Linux workstation to create a Butane file to fit your needs. Below is the GitHub for Bupy and a good example of running an app (Paperless NGX) on FCOS with the CoreOS Layering features.<p><a href="https:&#x2F;&#x2F;github.com&#x2F;quickvm&#x2F;bupy">https:&#x2F;&#x2F;github.com&#x2F;quickvm&#x2F;bupy</a><p><a href="https:&#x2F;&#x2F;github.com&#x2F;quickvm&#x2F;fcos-layer-paperless-ngx">https:&#x2F;&#x2F;github.com&#x2F;quickvm&#x2F;fcos-layer-paperless-ngx</a><p><a href="https:&#x2F;&#x2F;coreos.github.io&#x2F;rpm-ostree&#x2F;container&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;coreos.github.io&#x2F;rpm-ostree&#x2F;container&#x2F;</a><p><a href="https:&#x2F;&#x2F;github.com&#x2F;coreos&#x2F;enhancements&#x2F;blob&#x2F;main&#x2F;os&#x2F;coreos-layering.md">https:&#x2F;&#x2F;github.com&#x2F;coreos&#x2F;enhancements&#x2F;blob&#x2F;main&#x2F;os&#x2F;coreos-l...</a><p><a href="https:&#x2F;&#x2F;github.com&#x2F;coreos&#x2F;layering-examples">https:&#x2F;&#x2F;github.com&#x2F;coreos&#x2F;layering-examples</a>

What these sort of introductions to immutable always fail to consider is the other side of the coin, image-based. I&#x27;m working on <a href="https:&#x2F;&#x2F;universal-blue.org&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;universal-blue.org&#x2F;</a> along with many people much more skilled than me. We build OCI container images on top of vanilla Fedora Silverblue &amp; many other editions with different desktops. Those images can then be booted to (or rather rebased to) using rpm-ostree. This is a more robust way of extending the system than layering, and the same changes can be easily benefited or inherited from by anyone. You can even make your own image really easily!<p>I think that VanillaOS and SUSE are working on similar things, but we&#x27;re not an OS project, just a downstream from Fedora. Fedora&#x27;s full support is underway but with what&#x27;s already working perfectly our methods are already IME some of the most robust and easy ways of delivering Nvidia drivers for example.

I&#x27;m less interested in immutable systems and more in pre-configured systems. NixOS with Home Manager is the one that stands out here, but the configuration is just awful. I want to be able to have my full config in source control and know that is the state of my current system, with anything else being wiped on reboot. Anything that&#x27;s changed before reboot should be highlighted.<p>In my (limited) experience with something like Silverblue, the base system can be configured but when you start adding applications (like say, Firefox), it is lacking when it comes to configuring that because you&#x27;re using Flatpak and I don&#x27;t know of a way to tell it to both install all Flatpaks I want along with all of the configuration.<p>I guess there&#x27;s some way of installing Flatpaks en masse and then dotfiles can take care of the rest?<p><a href="https:&#x2F;&#x2F;universal-blue.org&#x2F;tinker&#x2F;mindset&#x2F;#resist-the-urge-to-add-the-entire-universe" rel="nofollow noreferrer">https:&#x2F;&#x2F;universal-blue.org&#x2F;tinker&#x2F;mindset&#x2F;#resist-the-urge-t...</a>

The problem I had with flatpak and the immutable approach in general is that I can&#x27;t modify them in ways that aren&#x27;t supported by the developer. For example, I use decsync to sync my calandars but as far as I can tell, it&#x27;s impossible to add the decsync plugin to the evolution flatpak.<p>Until these immutable systems support the stacking of custom overlay filesystems as a first class feature^1, people will continue to run mutable systems.<p>1: to account for use cases the developer can&#x27;t or won&#x27;t support.

I think the definition should be:<p>Installing any number of packages, then removing them in any order, at any future point(s) in time, is equivalent to never having installed them at all.<p>This leaves some distros out, but I feel like it’s the important part of the concept.

Been using Fedora Silverblue since its release and it&#x27;s absolutely the future. ostree is what everyone should be using.

I got into Tinycore this summer. Useful complement to the security philosophy &quot;One OS, one function&quot; which is kinda the thing behind Qubes, Tails and Whonix we talked about here a few days ago.<p>It&#x27;s so light, you can spin up VMs, one for a mail-server, one for a database, one for a firewall&#x2F;router, each in a couple of seconds.<p>Tinycore is itself immutable, so you add a vdisk with a &quot;package&quot; and some config, mark it read-only, and job done. A single Virsh script handles the startup and shutdown of &quot;services&quot; - each being a Tinycore instance. Fun, and robust so far, but not sure if I&#x27;d put it into anyone&#x27;s production just yet.

I have been using Fedora Sericea since it came out (it&#x27;s basically Fedora Silverblue, but uses Sway-wm instead of Gnome-wm). The system is actually pretty usable, and you don&#x27;t really need to reboot after each rpm-ostree install command (`rpm-ostree live-apply` takes care of it via systemd-based overlay).

&gt; immutability is a lie, many parts of the systems are mutable, although I don&#x27;t know how to describe this family with a different word (transactional something?).<p>In the case of Nix, it sounds like it&#x27;s more focused on reproduce-ability? It sounds like I should be able to take the Nix configuration file, plop it on another computer, and get the same system (except, perhaps, for &#x2F;home).<p>Some of the others sound more like existing tools that provide snapshot&#x2F;rollback capability, just with different implementations.

On the server-side, there&#x27;s Bottlerocket OS [1] (Amazon). They use A&#x2F;B partitions for upgrades, and the idea is that you just run containers for anything non-base. Boot containers are used to do custom configuration at boot, and host-container (or DaemonSet, if you run K8S) is used for long-running services.<p>[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;bottlerocket-os&#x2F;bottlerocket">https:&#x2F;&#x2F;github.com&#x2F;bottlerocket-os&#x2F;bottlerocket</a>

<p><pre><code> &gt; 4.3. Facts § &gt; - NixOS &#x2F; Guix are doing it right in my opinion</code></pre>

Relatedly, does anyone know if the security guarantees around distrobox have gotten any stronger? Last I looked, they promised nothing, but curious if there has been any movement there.<p>I would love if there was a seamless way to launch a distrobox os with a separate user home that could not touch my host system.<p>Likely a Real Hard Problem, but even some isolation would probably be an improvement of running everything under the same user account.

How does working with Docker work on Immutable systems like Fedora Silverblue. Like e.g. developing an application (in a Devcontainer like e.g. Toolbox to avoid having to install all the devtools on os-tree) and then building and debugging a Docker container from within the devcontainer? Or am I thinking in a wrong way?<p>Any good blogposts on developer workflows on Silverblue?

It&#x27;s interesting to watch these immutable images being adopted in the wider computing community. This has been a thing forever in the embedded world with Yocto&#x2F;Peta Linux&#x2F;Buildroot images. The image is a usually a straight disk image and is read-only in operation. This doesn&#x27;t inherently fix all the insecure IOT stuff as you usually need some way to reimage the device with updates and it takes at least some skill to do the bootloader signature verification right. It does help though as well as keep things deterministic.

Another really nice Immutable Linux system that I&#x27;m using is VyOS.. It&#x27;s targeted primarily at a router OS, but you can run containers on it now to make it pretty versatile.<p>Basically, it&#x27;s an image based OS that configures everything from a single config file on boot. <a href="https:&#x2F;&#x2F;docs.vyos.io&#x2F;en&#x2F;latest&#x2F;introducing&#x2F;about.html" rel="nofollow noreferrer">https:&#x2F;&#x2F;docs.vyos.io&#x2F;en&#x2F;latest&#x2F;introducing&#x2F;about.html</a>

The initramfs-overlay package became a trivial install with OverlayFS being accepted into the kernel a few years back.<p>This meant the mess systemd created each boot, could be dropped into the ram-drive with zero impact on the OS image. Effectively turning any Debian based system into a read-only OS backing image, but retaining the ability to boot into a normal writable system with a single boot flag.<p>This trick is a lot less finicky these days. =)

Good timing, I&#x27;m minutes away from installing openSUSE Aeon.<p>MicroOS Desktop turned into Kalpa (KDE) and Aeon (GNOME), but the latter has all the momentum.

It don&#x27;t understand what&#x27;s the big deal about it. To make the system immutable, isn&#x27;t it enough during boot to just<p><pre><code> 1. create a ram-backed filesystem 2. copy `&#x2F;`s contents to that new filesystem 3. Optional: Unmount `&#x2F;` 4. Mount the ram-backed filesystem on `&#x2F;` </code></pre> ?<p>You can easily do that from the initramfs during booting. I&#x27;m applying this patch to the roofs created by debootstrapping Debian Buster. You can use the system just fine and make changes as you please. But when you shut it down, it&#x27;s all lost. Everything I want to keep (like the permanent storage this system makes available over sshfs, NFS) is on seperate disks anyway. Sure, you need enough RAM to hold the entire rootfs (1.2G in case of Debian Buster) and it increases boot time a bit. For server applications, I don&#x27;t care at all.<p><pre><code> --- a&#x2F;usr&#x2F;share&#x2F;initramfs-tools&#x2F;scripts&#x2F;local 2021-11-05 12:50:23.541088057 +0100 +++ b&#x2F;usr&#x2F;share&#x2F;initramfs-tools&#x2F;scripts&#x2F;local 2021-11-05 13:02:14.483203576 +0100 @@ -180,9 +180,20 @@ # Mount root # shellcheck disable=SC2086 - if ! mount ${roflag} ${FSTYPE:+-t &quot;${FSTYPE}&quot;} ${ROOTFLAGS} &quot;${ROOT}&quot; &quot;${rootmnt?}&quot;; then - panic &quot;Failed to mount ${ROOT} as root file system.&quot; - fi + #if ! mount ${roflag} ${FSTYPE:+-t &quot;${FSTYPE}&quot;} ${ROOTFLAGS} &quot;${ROOT}&quot; &quot;${rootmnt?}&quot;; then + # panic &quot;Failed to mount ${ROOT} as root file system.&quot; + #fi + + mkdir --parents &#x2F;tmp&#x2F;diskroot + mount -t ${FSTYPE} ${roflag} ${ROOTFLAGS} ${ROOT} &#x2F;tmp&#x2F;diskroot + + mount -t tmpfs -o size=6G none ${rootmnt?} + chmod 755 ${rootmnt} + + cp --force --archive --verbose &#x2F;tmp&#x2F;diskroot&#x2F;* ${rootmnt} + + umount &#x2F;tmp&#x2F;diskroot + rm -r --force &#x2F;tmp&#x2F;diskroot } local_mount_fs()</code></pre>

I feel pretty strongly against the idea of immutable infrastructure when you&#x27;re &quot;infrastructure&quot; (shared systems, running other people&#x27;s software), but this article isn&#x27;t about that.<p>In my observation (and in datasets that I have access to), computers systems tend to follow the &quot;infant-mortality&quot; curve. This means that if they run for a little bit, they&#x27;re likely to run for a long time (and in addition, if you have many of them, they tend to die around the same time). My conjecture is that many computer systems have initialization routines which are not as thoroughly tested as the normal operating state of the system. Due to this, we tend to run into more issues in &quot;immutable&quot; systems than you otherwise would in &quot;mutable&quot; systems.

Off topic, but does anyone know how to find out where mutable data for NixOS modules are stored at (e.g. the data directory for a database) without reading the source? Occasionally, it&#x27;s mildly annoying, and would be comforting to know with certainty where all my state is.

I&#x27;m really, really happy with my current setup of Fedora immutable + toolbox [0]. This tool lets you create rootless containers that are fully integrated with the system, so you have acces to a regular mutable system, can install whatever without layering on the base system, run graphical apps, etc. while still having everything inside a container in your home directory. That means no Flatpak required. Highly recommended.<p>[0] <a href="https:&#x2F;&#x2F;containertoolbx.org" rel="nofollow noreferrer">https:&#x2F;&#x2F;containertoolbx.org</a>

Our team from Triton DataCenter &amp; SmartOS[1] is also working on an immutable Linux distribution[2] based on Debian + ZFS + LXC.<p>Currently this is supported on Triton DataCenter only, but our internal roadmap has us building a standalone version similar to how folks use SmartOS standalone.<p>[1]: <a href="https:&#x2F;&#x2F;www.tritondatacenter.com&#x2F;smartos" rel="nofollow noreferrer">https:&#x2F;&#x2F;www.tritondatacenter.com&#x2F;smartos</a><p>[2]: <a href="https:&#x2F;&#x2F;github.com&#x2F;TritonDataCenter&#x2F;linux-live">https:&#x2F;&#x2F;github.com&#x2F;TritonDataCenter&#x2F;linux-live</a>

Solene&#x27;s blog is great overall

Since we’re here, does anyone have a good solution for compiling PyInstaller binaries for different Linux distros? Would it be sufficient, say, to use a Ubuntu 16 compiled binary on Ubuntu 18, 20, etc. Presently we use different Docker images for each target distribution, however this is very resource intensive. Our target distros are several versions of Ubuntu, Centos, RockyLinux and Debian.

Glad to see EndlessOS included!<p>Silverblue&#x2F;Sircea gets all the attention these days, but Endless is the oldest OSTree-based user distro by a long shot, and it’s still actively developed by the Endless Foundation.<p>It’s also the one most suitable for non-technical users. Definitely worth considering for that use case, particularly for very young users since it now includes plenty of tutorial content intended for that audience.

The pros section feels a little light. The reason an immutable OS is attractive is to be able to cleanly remove files&#x2F;folders from the system.

I am currently on Mint, but will soon switch to Fedora Silverblue again, and then I encapsulate things for dev with Distrobox, which also can encapsulate home directory and export apps to the host system, see <a href="https:&#x2F;&#x2F;youtu.be&#x2F;Q2PrISAOtbY" rel="nofollow noreferrer">https:&#x2F;&#x2F;youtu.be&#x2F;Q2PrISAOtbY</a><p>That&#x27;s basically what I want, encapsulation and buttons :)

How would you approach making immutable live-cd like Linux? No persistence at all, just boot it and run some app - think some kind of presentation panel which shows predefined program&#x2F;URL. Ideally net booted to avoid having storage at all.

Is anyone running diskless Alpine in production? It seems optimal to me but extremely uncommon.<p>In the past I’ve tried running a small USB drive in rented bare metal w&#x2F; diskless alpine, but the machine seemed to reboot randomly IIRC.

Wait, I thought the reason sip on macOS was so terrible was because you couldn’t overwrite whatever system files you wanted without jumping through hoops… why would you bring an anti feature like that to Linux?

If you ran root on ZFS with a single mount &#x2F; form, and snapshotted it immediately after install, would that count as &quot;immutable&quot; in this logic?

There is also this: <a href="https:&#x2F;&#x2F;stal-ix.github.io&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;stal-ix.github.io&#x2F;</a><p>Fully statically linked.

There is Ubuntu Core too.<p><a href="https:&#x2F;&#x2F;ubuntu.com&#x2F;core" rel="nofollow noreferrer">https:&#x2F;&#x2F;ubuntu.com&#x2F;core</a>

Does MX Linux Frugal count as immutable?

&quot; system upgrades aren&#x27;t done on the live system packages changes are applied on the next boot you can roll back a change<p>Depending on the implementation, a system may offer more features. But this list is what a Linux distribution should have to be labelled &quot;immutable&quot; at the moment.&quot;<p>Immutable. I do not think it means what you think it means.

No mention of proprietary drivers such as nVidia&#x27;s? §

&gt; 4.1. Pros §<p>&gt; 4.1.1 you can roll back changes if something went wrong.<p>&gt; 4.1.2 transactional-updates allows you to keep the system running correctly during packages changes.<p>Last time I needed to roll back was OpenSSL in Ubuntu 18.04 and that was on one system.<p>I don think I&#x27;ve ever had a problem that 4.1.2 solves.<p>I don&#x27;t want to have yet another Linux OS to solve a problem that happens once a decade.<p>If I wanted an immutable OS, I&#x27;d use a container OS on which I&#x27;d run apps as containers. Oh, wait, I already have that.

Why are we still talking about broken notions of immutable systems when we actually have trusted execution environments and secure boot and cryptographically sealed strong assurances of what is executing at any point and can do secure upgrades, even with remotely attached secure storage and do it across a huge fleet of machines at very large scales?