Why systemd is a problem for embedded Linux

I couldn&#x27;t disagree more: I&#x27;ve worked with lots of embedded devices running systemd, and it solves many more problems than it introduces. The community is also quite responsive and helpful in my experience.<p>I won&#x27;t pretend there aren&#x27;t occasional weird problems... but there&#x27;s always a solution, here&#x27;s a recent example: <a href="https:&#x2F;&#x2F;github.com&#x2F;systemd&#x2F;systemd&#x2F;issues&#x2F;34683">https:&#x2F;&#x2F;github.com&#x2F;systemd&#x2F;systemd&#x2F;issues&#x2F;34683</a><p>Memory use is irrelevant to me: every embedded Linux device I&#x27;ve been paid to work on in the past five years had over 1GB of RAM. If I&#x27;m on a tiny machine where I care about 8MB RSS, I&#x27;m not running Linux, I&#x27;m running Zypher or FreeRTOS.

OpenEmbedded&#x2F;Yocto, Devuan and Gentoo provide multiple init systems.<p>systemd CVEs: <a href="https:&#x2F;&#x2F;ubuntu.com&#x2F;security&#x2F;cves?package=systemd&amp;limit=100" rel="nofollow">https:&#x2F;&#x2F;ubuntu.com&#x2F;security&#x2F;cves?package=systemd&amp;limit=100</a><p>Rust PoC: <a href="https:&#x2F;&#x2F;github.com&#x2F;KillingSpark&#x2F;rustysd">https:&#x2F;&#x2F;github.com&#x2F;KillingSpark&#x2F;rustysd</a><p><i>&gt; Rustysd is a service manager that tries to replicate systemd behaviour for a subset of the configuration possibilities. It focuses on the core functionality of a service manager, not requiring to be PID1 (aka init process).. core systemd functionality is not very hard to replicate.. the advantages of having a systemd-like service manager can be brought to many other platforms.. (like alpine linux, commonly used in docker containers and small vms) and freebsd.</i>

&gt; The more fundamental problem is that the people who most like systemd are distribution managers. (...) a distribution manager doesn’t have to maintain, integrate, and support a whole bunch of system utilities from different sources – systemd provides everything in one huge build.<p>That’s actually a great lesson in optimizing your product for adoption.

I think what he&#x27;s running into is &quot;the unix philosophy&quot;.<p>It is basically about small tools, limited in scope, that solve a problem surgically and comprehensively. They work together by mixing and matching to solve the problem.<p>systemd is sort of like if microsoft word took on system init. In the beginning it was small, but now it is web based and does videoconferencing.<p>I don&#x27;t know, it is sort of like busybox, replacing other system components in a limited Minimum Viable Product way, but more like limited function, not limited resources.

I&#x27;m conflicted on the name&#x2F;issue. I don&#x27;t have any issues with Linux on embedded systems, because I&#x27;d use a minimal init with that one single app it should be running. (Or as mentioned in other comment, build your minimal distro yocto-style) RPi is a low spec general computer at this point. The idea of running gnome on an embedded system just doesn&#x27;t make sense in my mind...<p>RPi took the form factor that was used for industrial control boards and made it into a usable desktop... which people use as a desktop (or home server). But it created a segment which really is neither. And a lot of the software just isn&#x27;t written with that segment in mind. Maybe as it gets more popular, people will pay more attention. But currently it seems more of an expectations issue.

<a href="https:&#x2F;&#x2F;chimera-linux.org&#x2F;" rel="nofollow">https:&#x2F;&#x2F;chimera-linux.org&#x2F;</a> <a href="https:&#x2F;&#x2F;chimera-linux.org&#x2F;docs&#x2F;faq#what-is-the-projects-take-on-systemd" rel="nofollow">https:&#x2F;&#x2F;chimera-linux.org&#x2F;docs&#x2F;faq#what-is-the-projects-take...</a> Is going a really interesting direction with it&#x27;s service management and login &#x2F; seat management. Fwiw, I think it&#x27;s author also has a more gounded perspective on systemd then many others. For Chimera the main issue is systemd&#x27;s high volume use of gnulibc and gcc extensions, which is a porabiluty issue for Chimera which uses a LLVM toolchain and musl libc (and a bunch of really cool hardening and compiler engineering)

I’ve never used it as an init replacement, but runit (<a href="https:&#x2F;&#x2F;smarden.org&#x2F;runit&#x2F;runit.8" rel="nofollow">https:&#x2F;&#x2F;smarden.org&#x2F;runit&#x2F;runit.8</a>) is great for supervising applications and services.<p>- It’s built into BusyBox and is very lightweight - Configuration is dead simple (everything is basically shell scripts in the filesystem; no special config language) - Includes logging functionality with rotation - Easily controllable from other applications using named pipes - It’s an almost perfect embodiment of the Unix philosophy: a series of small, single-purpose executables (svlogd, runsv, runsvdir, etc), with no strange config file formats or protocols.<p>Regarding udev, it’s entirely possible to run an embedded system without it if your device has a fixed set of peripherals, doesn’t need to handle hotplugged hardware, and uses a fixed set of kernel modules. In Yocto, you can define a dummy device manager recipe and boom, no udev. As a bonus, removing udev can shave several seconds off your boot time.

Wait so systemd is a problem for embedded Linux because... it uses 250MB of RAM? So it&#x27;s only really a problem for systems where using a mainstream distro probably wouldn&#x27;t even be a consideration anyway (yocto seems to be popular, as mentioned above). I&#x27;m not seeing a strong argument here. At the risk of inviting a parade of &quot;actually, one time at band camp I saved the day with a first-gen raspberry pi zero&quot;, how many embedded systems in 2024 are really RAM-limited?

The memory comparison isn’t really fair as SysVInit relies on external utilities for starting and stopping processes.<p>At the very least, SysVInit requires a shell for running init scripts which systemd doesn’t need for native service files.

&gt; I wish I knew the solution to this problem<p>Ooh ooh, I know! It&#x27;s to improve systemd for the use-cases outlined and build a really great open source init&#x2F;process manager.

I have no philosophy issue with systemd. The biggest show-stopper (on embedded Linux) for me is its install size. We have edge devices with only 64M RAM and 128M NAND flash. For an OTA update (via Bluetooth sometimes) only ~20M in total, adding ~5M systemd plus its dependencies doesn&#x27;t make sense. 5MiB is roughly equivalent to the entire Linux kernel image on those devices.<p>Among the components of systemd, the part that has the lowest added value for our use is dbus. It&#x27;s designed primarily for desktops and useless for most of our systems, but unfortunately, it is also a hard requirement for systemd.

The BSDs may be more to your liking. They&#x27;re a lot more minimalist than Linux now.

&gt; With care, though, a minimal installation of systemd does run on a low-resource ARM board like the Pi 3. In fact, it will run on a board with 1Gb RAM, perhaps even lower.<p>Come fucking on. I ran systemd on a 32Mb MIPS board (RouterBoard from Mikrotik) just fine.<p>The RES column is incredibly misleading in this case, because systemd is the first app in the system. So it gets charged for loading the entire glibc and the supporting cast of libraries.<p>If you want to look at the true usage, check the RssAnon value:<p>&gt; root@kmipsrv:&#x2F;proc&#x2F;1# cat &#x2F;proc&#x2F;1&#x2F;status | grep RssAnon<p>&gt; RssAnon: 3220 kB<p>You can also pare this down a bit.<p>Similarly for journald:<p>&gt; root@kmipsrv:&#x2F;proc&#x2F;1# cat &#x2F;proc&#x2F;231&#x2F;status | grep RssAnon<p>&gt; RssAnon: 640 kB<p>So in reality, systemd works just fine for the vast majority of embedded platforms. If you can spare 3Mb of RAM, then you can run a full-blown dependency management system with reliable recovery, logging integration, and event-driven device management. Soon with support for verified boot with hardware root-of-trust.

I can only say that the obvious answer here is to throw effort into supporting &amp; improving Gentoo.

I&#x27;ve had very nice experiences using systemd for developing embedded systems. Laying out all the unit files and just slapping the files in an overlay makes it super simple.<p>The RAM usage isn&#x27;t being calculated correctly. You can easily run systemd on very limited systems.

Try s6 or runit, and mdev from busybox. Mdev is not a &quot;full featured&quot; udev replacement, but that&#x27;s sort of the point. It does just enough to get the job done and is extensible with scripts.<p>If you want to do the size of embedded where this matters, you also probably want a smaller libc like musl, and then buildroot and busybox are your friends.

I run systemd-based Debian just fine on 512 MB RAM BeagleBoneBlack SBCs with 4GB emmcs. Now emacs, that&#x27;s a problem due to the large install size.

If it can run X11 on a directly connected display, it&#x27;s probably not embedded spaces ;). Don&#x27;t take this comment too seriously, however raspberry pi is definitely at the &quot;more desktop-like&quot; end of the embedding pool.<p>As for systemd, it&#x27;s an interesting set of tools, and one can if one wishes pick which ones are active in a system, if that system is complex enough to be running Linux (or other compatible OS) to begin with.

The article fails to mention another aspect of systemd that interferes with emmbedded dvelopment: it&#x27;s utility applications (primarily systemctl, but all the others as well) are intended to be run on the booted system for which the control is being performed.<p>When configuring boot media, like an SD card, for an embedded ssystem that is not the running system where the configuration is occurring, this is an impediment. There is systemd-firstboot, etc, but this is not as convenient as just being able to set config options on the mounted (non-booted) embeddded media.<p>I&#x27;ve never liked it, I still don&#x27;t like it, and I think the number of people in this camp is understated by the article.<p>However, I am still running it. As other posts have mentioned, switching distributions is a major hasstle, especially if you&#x27;ve built tools using the distro&#x27;s architecture. For me, this is archlinux.<p>Although, I am in the process of testing and migrating to void linux. Which is systemd free, and hosts ARM and x86 binary package repositories.

As for a solution to this, wouldn&#x27;t it be possible to use systemd to build a targeted init script? Use the expertise embedded in systemd to bring up the system you&#x27;re targeting, recording how it does that and then removing systemd, leaving the artifacts?<p>Why keep this huge monolithic system running in the background after it has served its purpose?<p>Also, with ai, would it be possible to redraw the lines within systemd between the separate domains&#x2F;concerns, then redistribute the functions back to where they conceptually belong? Then you can refine those changes to make them more standardized and universal instead of interdependent and centralized.

We (<a href="https:&#x2F;&#x2F;github.com&#x2F;husqvarnagroup&#x2F;smart-garden-gateway-public">https:&#x2F;&#x2F;github.com&#x2F;husqvarnagroup&#x2F;smart-garden-gateway-publi...</a>) are using systemd on devices with 128 MB of RAM and are happy with it. It solves so many problems one usually has to fiddle with, totally worth a few MBs of RAM.

You could checkout void and alpine.

I&#x27;m quite annoyed by the bland statement in the introduction of this article: people make about it don’t stand up to much scrutiny, even the technical ones<p>Basically the author says that everyone else has stupid rants that doesn&#x27;t stand but himself has valid claims so that in the end we have a proof that there are a few valid technical rants!<p>That being said, the main problem with systemd from the beginning is that it is a cancer. It was explicitly designed to prevent as much as possible the modularity and intercompatibility. Kind of opposite to the POSIX way even I think.<p>Despite what is pretended, doing the things would probably not have impacted features, like boot speed but it was key for them to shove through our throat the usage of systemd in a &quot;all or nothing&quot; way.<p>And udev or logind are good example of that.

It is unfortunate that after all this time the basic reasons why systemd was introduced and has been so successful are pervasively misunderstood.<p>&gt; The majority of Linux users are uninterested in the pros and cons of systemd. A small number are violently opposed to it, and a small number are violently opposed to those who are opposed to it.<p>Starting out by framing systemd in terms of opposition deliberately steers attention away from the issues that drive its adoption. The tone is moderate, but this is not good faith argumentation.<p>&gt; I think there’s little argument that the main target for systemd is a general-purpose computer, with a modern, integrated graphical desktop<p>That is one of the targets. The main target is servers that run the web of services typically required to back up modern applications. What if an application depends on a resource server which itself depends on a remote filesystem being mounted and an event service running. Theoretically that can work with System V init, but how tricky is that to put in place and make robust? Administrators for systems vending complex interdependent services are the main target and always have been.<p>&gt; Unfortunately, what makes systemd good for general-purpose desktop applications<p>Get it wrong and then run with it. Failing to understand what is driving adoption of systemd leads to arguments about it that don&#x27;t really go anywhere.<p>&gt; The more fundamental problem is that the people who most like systemd are distribution managers. Sure, there are administrators who like it, and defend it vigorously; but most end users and administrators don’t really care.<p>Not distribution managers, but system administrators. They are the ones with the complex needs that systemd serves. As such they advocate for it and because of the lack of working alternatives distribution managers switch to it in order to please the largest number of clients. Thinking of systemd in terms of desktop systems and distribution managers means misunderstanding why it is there in the first place.

I used it heavily at work. Then moved to a shop that does exclusively containers and realized how incredibly immature engineers’ understanding of the problem space is.<p>Container startup ordering, startup dependency management, socket handling and all things timing and network related are frankly a complete mess in modern infrastructure. Even, surprisingly in well maintained projects like k8s. I find myself shaking my head at problems solved by systemd a decade ago far too often.

TBH, my biggest gripe is that systemd doesn&#x27;t play nice with musl or ulibc, I would like to not have to ship a cron, a supervisor, a xinetd equivelant, and a binary just to react to inotify&#x2F;dnotify events. Though openrc how has a supervisor.

Big systemd on embedded devices is actually A GOOD THING! Imagine what Gentoo, Buildroot&#x2F;OpenWrt or even Busybox alone could do with so much memory after removing all that bloated OS!

Excellent plan, go to the other inits, bug bounties are what I do over christmas and init systems and hand rolled init scripts are a great source of security flaws.<p>Taking notes on which distros dont run systemd, for when I want to make some money over christmas.

For me as a user, systemd is far better than old init systems. The logging alone is worth it to switch to it. Add more ram to the embedded board and welcome to the future.

I think author is missing on the Devuan-maintained udev fork.

Not really embedded, but one of the easier classes of wins I&#x27;ve had over the years is moving prod servers to a non-systemd Gentoo. Random (rare) 200ms latency spikes and whatnot just disappear, and the system jitter drops down to the tens of microseconds range, significantly improving the qualitative performance of basically every piece of software I care about.

I like systemd sometimes. Other times it confuses me. Particularly the way cgroups work. The way splicing works. Why oracle mem usage is still uncontrollable when I set user splice restrictions. Idk, I’m still learning it. I don’t like the terminology it uses sometimes too like “snapshot.” When I first read about systemd snapshot I was like oh cool you can snapshot the system. No. Also some esoteric things it supposedly supports don’t work sometimes. I find the idea of systemd targets and fstab having anything to do with each other anathema. Idk.

Meh, we are not in 2021 or 2018 anymore.<p>Eudev and elogind ARE mantained and used in voidlinux and gentoo, and anyways being that embedded it doesn&#x27;t seem clear to me where would the &quot;hard dependency of a very essential software on some systemd-thingd&quot; happen.

&quot;I&#x27;m using a thing in a way it&#x27;s not meant to be used and it sucks&quot;<p>Is this even a serious post.

This is anti-systemd FUD. Implying that applications which work without systemd are going to stop doing so. Suggesting that Linux distributions adopted systemd because it&#x27;s good for distributors and not because users actually want it. We are &quot;forced&quot; to use systemd&#x27;s udev implementation. Please.

There is another aspect to the memory footprint: Does systemd fit into the L1 cache? Does sysvinit?<p>Both x86 and Linux (page cache) rely on caches for performance.

Systemd, Wayland, DBus, and other modern Linux &quot;solutions&quot; are all problematic for the same reason: they are complex monoliths that are extremely difficult to replace, in part or in whole. Systems often won&#x27;t work without them, because the entire ecosystem has grown dependent on them. You literally can&#x27;t run a modern desktop without a bunch of &quot;shims&quot; whose sole purpose is to <i>fake</i> being systemd.<p>The best designs, sadly, don&#x27;t win out. The ones that win out are the ones that serve the interests of the largest players. And the largest players are usually RedHat, Google, and other vendors, who just want to get &quot;their&quot; needs met and move on. They couldn&#x27;t give a fig about compatibility with some other random tiny project, or an embedded device. And those people working on tiny projects aren&#x27;t in the room when &quot;standards&quot; are decided for the entire community.<p>There are some hold-outs. Slackware went a long time without it, but next release will unfortunately use systemd. I&#x27;ve run Alpine Linux (systemd-free, so far) as my laptop OS for years. I&#x27;ll probably switch to another distro that&#x27;s more user friendly, is easier to contribute to. But it&#x27;s embarrassing and annoying that we have to pretend to be systemd-compatible just to run a desktop. We should call it &quot;GNU&#x2F;Systemd&#x2F;Linux&quot; since it&#x27;s now one big monolith.