Why and how we’re migrating many of our servers from Linux to the BSDs

I use Linux, FreeBSD, NetBSD, and OpenBSD all for fun, learning, and profit(the first two).<p>At the very least it is nice to make acquaintance with at least one BSD because it will probably expand your knowledge on Linux in ways you wont be able to anticipate.<p>For example, FreeBSD got me into kernel development, full system debugging, network stack development, driver development, and understanding how the whole kit fits together. Those skills transferred back and forth with reasonable fidelity to Linux, and for me jumping into Linux development cold would have been too big a leap.. especially in confidence and developing a mental model.<p>For my personal infrastructure, I tend to use FreeBSD because in many ways it is simpler and less surprising, especially when accounting for the passage of time. ifconfig is still ifconfig, and it works great. rc.d is all I need for my own stuff. I like the systematic effects of things like tunables and sysctl for managing hardware and kernel configuration. The man pages are forever useful to new and old users. The kernel APIs and userland APIs are extremely stable akin to commercial operating systems and unlike Linux.<p>There are warts. There are community frictions. The desktop story and some developer experiences will be perpetually behind Linux due to the size of the contributor base and user base. The job market for BSD is very limited compared to Linux. But I don&#x27;t think it&#x27;s an all or nothing affair, and ideally in a high stakes operation you would dual stack for availability and zero-day mitigation (Verisign once gave a great talk on this).

&gt; The largest failure was with btrfs — <i>after a reboot, a 50 TB filesystem (in mirror, for backups) simply stopped working. No more mounting possible.</i> Data was lost, but I had further backups. The client was informed and understood the situation. Within a few days, the server was rebuilt from scratch on FreeBSD with ZFS — since then, I haven’t lost a single bit.<p>As someone who admins a lot of btrfs, it seems very unlikely that this was unrecoverable. btrfs gets itself into scary situations, but also gets itself out again with a little effort.<p>In this instance “I solve problems” meant “I blow away the problem and start fresh”. Always easier! Glad the client was so understanding.

I like the idea and would like to learn more, but it looks like &quot;migrated stuff without testing ahead of time and it turned out faster for some reason&quot;. Was it the memory allocations? Was it the disk latency? The hypervisor? Could it be replicated by other means? It was a fun read, but the reasoning&#x2F;understanding was missing. I hope people investigate deeper before making changes like that.<p>If you look for benchmarks comparing databases on Linux&#x2F;BSDs you&#x27;ll find lots of nuance in practice and results going both ways depending on configuration and what&#x27;s being tested.

&gt; “If nothing is working, what am I paying you for? If everything’s working, what am I paying you for?”<p>Bloke is not acquainted with Keynesian economics.<p><a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=9OhIdDNtSv0" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=9OhIdDNtSv0</a><p><a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=NO_tTnpof_o" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=NO_tTnpof_o</a><p>All a man needs is food in his stomach and a place to rest at the end of the day. Everything else is vanity<p>What proportion of global GDP is dedicated to fulfilling our basic material needs?<p>It is mostly unnecessary. Inspite of the huge productivity gains made since the seventies, the current generation of young Americans are poorer than their parents and grandparents were at their age.<p>So what does all the IT optimization bring? Just more wealth for the owners and redundancies for their employees, including Joe Bloggs here.<p>It is time people in IT got to understand this. In the long term their activities are not going to improve their wealth. They are one of the few professions whose job is to optimize themselves out of a living, unless they own the means of the production their are optimizing, which they don&#x27;t.<p>It is their employers that do.

I switched from FreeBSD to Linux, mainly because of the bad Java support and the simple fact that Linux became way more popular, which added to the difference in software availability.

I&#x27;ve recently discovered systemd-nspawn which is an alternative to LXC, builtin and integrated into systemd. Much lighter than full VMs and it&#x27;s quite similar to Solaris Zones and FreeBSD jails. One way to use it is to extract an OCI (Docker) image to a path, that way you can reuse the container tooling provided by Docker, Podman et al.<p>I&#x27;ve barely touched the BSDs and it&#x27;s been a few years since I last used Solaris so I can&#x27;t make much of a comparison as a user myself.

I&#x27;ve become a fairly loyal OpenBSD user in the last 3-4 years. The base OpenBSD load includes a substantial amount of network capabilities, and is cleanly implemented. It&#x27;s almost too cleanly implemented, to the point of making me feel sort of guilty when I start to clutter up an install with a bunch of packages...<p>If my needs for storage were more complicated, I would probably use FreeBSD ZFS, but UFS suffices for my rather modest needs.<p>I use OpenBSD for desktop, web and mail services. There are some limitations, but none that are serious enough to warrant dealing with running another BSD, or Linux distribution.

I would differ on a small point. For SOHO usage, I think that docker compose is perfectly viable and often simplifies backup, migration and moving to a new server. Just my own take on this. A lot of apps really only need one instance with a good backup strategy and not hot failover instances and can handle an hour of down time once a year or two as needed, which I rarely experience.<p>As mentioned in the article, it also serves as a decent set of instructions, assuming the actual dockerfile(s) for the services and dependencies are broadly available. You can swap out the compose instance of PostgreSQL for your dedicated server with a new account&#x2F;db, relatively easily. Similar for other non-app centered services (redis, rabbitmq, etc). You can go all in, or partly in and in any case it does serve as self-documenting to a large degree.

I wish he could write up a bit about XFS failure he had. Im using it from many many years and there is no issues at all.

Glad to see all the major BSDs used here; I use OpenBSD whenever it makes sense.

I dont think the article is even about BSDs, but generally really bad things in the tech sectors and philosophy to tech.<p>&gt;my priority is solving my clients’ specific problems, not selling a predefined solution.<p>&gt;It’s better to pay for everything to work than to pay to fix problems.<p>&gt;computing should solve problems and provide opportunities to those who use it.<p>&gt;The trend is to rush, to simplify deployments as much as possible, sweeping structural problems under the rug. The goal is to “innovate”, not necessarily improve — just as long as it’s “new” or “how everyone does it, nowadays.”<p>&gt;Some people are used to thinking that the ideal solution is X — and believe that X is the only solution for their problems. Often, X is the hype of the moment<p>&gt;When I ask, “Okay, but why? Who will manage it? Where will your data really be, and who will safeguard it?”, I get blank faces. They hadn’t considered these questions. No one had even mentioned them.<p>&gt;We’ve lost control of the data. For many, it’s unnecessary to complicate things. And with every additional layer, we’re creating more problems.<p>Hopefully someday more people will wake up.

All is fine and dandy and BSDs can solve many use cases. Unfortunately for the solution we are working on, which imply many microservices we need Kubernetes and no BSD equivalent to Kubernetes exists.

I love it when the answer to a question posed in a headline is provided in the second sentence of the article.<p>&gt; I’m the founder and Barista of the BSD Cafe, a community of *BSD enthusiasts<p>Did the original article change it&#x27;s title (currently &quot;I Solve Problems&quot;), or did the submitter editorialize it?

FreeBSD is great as a server. Wifi performance still sucks. The author praised byhve but byhve is not all it’s cracked up to be. Both Xen and Linux virtualization perform better, VMware as well. I like FreeBSD, but the other day I found it still uses sendmail. Rc.conf is simple to use, and the ports system is great…I just feel the author was pushing his “X” solution. Hardware support is important as well. I’ve used BSD for a SAN, and FW. Would I use it for a virtualization host? No.

I&#x27;m surprised no one is talking about FreeNAS &#x2F; TrueNAS and their interesting history in this area in the comments.<p>There&#x27;s probably more collective writing about the various tradeoffs between Debian and FreeBSD in their forums and communities than anywhere else on the internet.<p>Personally I love ZFS and ZFS on root so much I can never go back to not having it. It&#x27;s a shame more cloud providers like DigitalOcean&#x2F;AWS&#x2F;etc. don&#x27;t offer it natively.

&gt; [users ..] began explicitly requesting “jails” instead of Docker hosts. They started using BastilleBSD to clone “template” jails and deploy them.<p>huh, were they running persistent docker containers and modifying them in-place? If that&#x27;s the case, they were missing the best part of Docker - the Dockerfile and &quot;container is a cattle&quot;. The power of Docker is there no ad-hoc system customization possible, it&#x27;s all in Dockerfile which is source-controlled and versioned, and artifacts (like built images) are read-only.<p>To go from this to all-manual &quot;use bastille edit TARGET fstab to manually update the jail mounts from 13.1 to 13.2 release path.&quot; [0] seems like a real step back. I can understand why one might want to go to BSD if they prefer this kind of workflow, but for all my projects, I am now convinced that functional-like approach (or at least IaaC-like one) is much more powerful than manually editing individual files on live hosts.<p>[0] <a href="https:&#x2F;&#x2F;bastille.readthedocs.io&#x2F;en&#x2F;latest&#x2F;chapters&#x2F;upgrading.html#revert-upgrade-downgrade-process" rel="nofollow">https:&#x2F;&#x2F;bastille.readthedocs.io&#x2F;en&#x2F;latest&#x2F;chapters&#x2F;upgrading...</a>

Running 3 different BSDs is not my idea of solving problems.

All I can say is experiences differ. I&#x27;m a long time Debian user, and now use FreeBSD for work. Both are far better than the proprietary competition, but I&#x27;d take Debian&#x2F;Linux over FreeBSD when building a random server.<p>To give but one example, I recently reported a bug when FreeBSD didn&#x27;t boot after upgrade from 13 to 14. Worse the disk format was somehow altered so when the reboot tried to boot off 13 due to zfs bootonce flag (supposedly a failsafe), it refused to boot for the same reason. I believe it&#x27;s due to a race condition in geom&#x2F;cam. The same symptoms were reported 6 years ago, but the bug report has seen no activity. Making your system irrecoverable without a rescue image and console access strikes me as pretty serious. He waxes lyrical about zfs, but it&#x27;s slower and more resource hungry than it&#x27;s simpler competition and it&#x27;s not difficult to find numerous serious zfs bug reports over the years. (But not slower than FreeBSD&#x27;s UFS, oddly. It&#x27;s impressively slow.) Another thing that sticks in my mind is a core zfs contributor saying it&#x27;s encryption support should never have been merged.<p>This sounds too disparaging because the simplicity and size of FreeBSD has its own charms, but the &quot;it&#x27;s all sunshine and roses&quot; picture he paints doesn&#x27;t ring true to me. While it&#x27;s probably fair to say stable versions of FreeBSD are better than the Linux kernels from kernel.org, and possibly Fedora and Ubuntu, they definitely trail behind the standard Debian stable releases.<p>Comparing FreeBSD to Debian makes throws up some interesting tradeoffs. On the one hand, FreeBSD&#x27;s init system is a delight compared to systemd. Sure, systemd can do far, far more. But that added complexity makes for a steep learning curve and a lot of weird and difficult to debug problems, and as FreeBSD&#x27;s drop dead simple &#x2F;etc&#x2F;rc.conf system proves most of the time the complexity isn&#x27;t needed to get the job done. FreeBSD&#x27;s jails just make more intuitive sense to me than Linux&#x27;s equivalent which is built on control groups. FreeBSD&#x27;s source is a joy to read compared to most I&#x27;ve seen elsewhere. I don&#x27;t know who&#x27;s responsible for that - but they deserve a medal.<p>On the downside - what were they thinking when they came up with the naming scheme under &#x2F;dev for block devices? (Who thought withering device names was a good idea, so that &#x2F;dev no longer reflects the state of attached hardware?) And a piece of free advice - just copy how Linux has does it&#x27;s booting. Loading a kernel + initramfs is both simpler and far more flexible then the FreeBSD loader scheme. Hell, it&#x27;s so flexible you can replace a BIOS with it.<p>The combination of the best parts Linux and the BSD&#x27;s would make for an wonderful world. But having a healthy selection of choices is probably more important, and yes - I agree with him that if you are building an appliance that has an OS embedded in it, the simplicity of FreeBSD does give it an edge.

&quot;As an experiment, I decided to migrate two hosts (each with about 10 VMs) of a client — where I had full control—without telling them, over a weekend.&quot; And that&#x27;s where I draw the line. Abusing the trust of your customers is an absolute no-no in my book.

Fidonet, it was been a while since I saw that.

One of the website providers I use (Pair) for 20+ years used to be exclusively FreeBSD. I believe they use a lot of Linux now. Not sure why.

I really wanted to give a try to FreeBSD... because I thought it was linux from &quot;better&quot; times... and then I saw they are tied to the tons of similar gcc extensions... and then I say to myself &quot;why bother since it has the same major compiler dependency issue&quot;, better try to fix linux code base or start from there.

Awesome writeup, thanks for that, it really puts the bsd&#x27;s in perspective of today&#x27;s tech industry. What&#x27;s the BSD version of k8s? You mention BSDs instead of k8s in the article.

If I want to run a BSD as just a file server - so I guess zfs + samba + bonjour or whatever the discovery protocol is these days - which BSD should I try?

Is BSD really significantly more efficient than Linux? The anecdotes here seem almost unbelievable.

the title of the talk is &quot;Why (and how) we’re migrating many of our servers from Linux to the BSDs&quot;<p>and that should be the title of this post too.<p>I like that the blog post shares the slides, not just the video.

&gt; As an experiment, I decided to migrate two hosts (each with about 10 VMs) of a client — where I had full control—without telling them, over a weekend.<p>Yeah. That guy should not be allowed anywhere near the production workloads. &quot;I solve problems&quot;, my ass.

U

Ladies and gentlemen, this person solves problems. Let it be known.