We're migrating many of our servers from Linux to FreeBSD

Funny enough, I decided to play with FreeBSD for personal projects in 2020. I gave up and I am reverting all my servers to Linux in 2022, for the opposite of the reasons mentioned in this article.<p>* Lack of systemd. Managing services through shell scripts is outdated to me. It feels very hacky, there is no way to specify dependencies, and auto-restarts in case of crashes. Many FreeBSD devs praise launchd, well... systemd is a clone of launchd.<p>* FreeBSD jail are sub-optimal compared to systemd-nspawn. There are tons of tools to create freebsd jails (manually, ezjail, bastillebsd, etc…) half of them are deprecated. At the end all of your jails end up on the same loopback interface, making it hard to firewall. I couldn&#x27;t find a way to have one network interface per jail. With Linux, debootstrap + machinectl and you&#x27;re good to go.<p>* Lack of security modules (such as SELinux) -- Edit: I should have written &quot;Lack of <i>good</i> security module&quot;<p>* nftables is way easier to grasp than pf, and as fast as pf, and has atomic reloads.

The reasons are, for a large part, not on the technical side. I was surprised, because this this a lot of work for little visible gain. Here are the reasons, slightly abbreviated:<p>&gt; The whole system is managed by the same team<p>Mostly philosophical.<p>&gt; FreeBSD development is less driven by commercial interests.<p>Mostly philosophical.<p>&gt; Linux has Docker but FreeBSD has jails!<p>IMO, this comparison is a mistake. In the Linux world, systemd&#x27;s nspawn is very similar to Jails. It&#x27;s a glorified chroot, with security and resource management. All the systemd tools work seemlessly with nspawn machines (e.g. `systemctl status`). Containers à la Docker are a different thing.<p>BTW, I thought the last sentence about security issues with Docker images was strange. If you care about unmaintained images, build them yourself. On the other side, the FreeBSD official documentation about Jails has a big warning that starts with &quot;Important: the official Jails are a powerful tool, but they are not a security panacea.&quot;<p>&gt; Linux has no official support for zfs and such<p>Fair point, though I&#x27;ve heard about production systems with zfs on Linux.<p>&gt; The FreeBSD boot procedure is better than grub.<p>YMMV<p>&gt; FreeBSD&#x27;s network is more performant.<p>Is there some conclusive recent benchmark about this. The post uses a 2014 post about ipv6 at Facebook, which I think is far from definitive today. Especially more since it &quot;forgot&quot; to mention that Facebook intended to enhance the &quot;Linux kernel network stack to rival or exceed that of FreeBSD.&quot; Did they succeed over these 8 years ?<p>&gt; Straightforward system performance analysis<p>The point is not about the quality of the tools, but the way each distribution packages them. Seems very very low impact to me.<p>&gt; FreeBSD&#x27;s Bhyve against Linux&#x27;s KVM<p>The author reluctantly admits that KVM is more mature.

&quot;Some time ago we started a complex, continuous and not always linear operation, that is to migrate, where possible, most of the servers (ours and of our customers) from Linux to FreeBSD.&quot;<p>I don&#x27;t really disagree with any of the stated reasons, but I also didn&#x27;t see a reason that would make me even consider making the move with our servers, or even bother with some small number of servers. At least for me, I&#x27;d need a bunch of REALLY GOOD reasons to consider a move like that. A huge cost savings AND some huge time savings in the future might do it.

The characterisation of systemd in this post really bothers me, particularly this:<p>&gt; 70 binaries just for initialising and logging<p>It’s just not true. Those 70 binaries provide much more functionality than an init system, they can cover a significant portion of system management, including a local DNS resolver, network configuration or system time management. You can dislike the fact everything is so tightly integrated (which feels ironic given that the post goes on to praise a user space from one team), but let’s at least be correct about this.

&gt; Linux has Docker, Podman, lxc, lxd, etc. but... FreeBSD has jails!<p>Docker, podman, lxc, lxd, etc are userland components. Linux has cgroups and namespaces.<p>FreeBSD jails are a bit more complicated because FreeBSD isn&#x27;t distributed the way Linux is. Linux is distributed as <i>just</i> the kernel, whereas FreeBSD is a base OS. This probably could&#x27;ve been phrased better as, &quot;Linux has no interest in userland and I want some userland consistency&quot;. That&#x27;s fair, Linux was built around the idea that operating system diversity was a good thing long term, FreeBSD was more interested in consistency. I&#x27;m reading between the lines, a bit, here because of the critique of SystemD (note: not all linuxes use SystemD)<p>Personally speaking, I like both Linuxes and FreeBSD but I don&#x27;t think debating the two is important. Rather, I&#x27;d encourage turning your attention to the fact that every other component on a system runs an OS-like interface that we don&#x27;t make open OS&#x27;s or &quot;firmware&quot; for.

&gt; Consider systemd - was there really a need for such a system? While it brought some advantages, it added some complexity to an otherwise extremely simple and functional system. It remains divisive to this day, with many asking, &quot;but was it really necessary? Did the advantages it brought balance the disadvantages?&quot;<p>This is really telling for the level of analysis done: systemd has been the target from a small number of vocal complainers but most working sysadmins only notice it in that they routinely deal with tasks which are now a couple of systemd stanzas instead of having to cobble together some combination of shell scripts and third-party utilities. Confusing noise with numbers is a dangerous mistake here because almost nobody sits around randomly saying “this works well”.

Linux took many markets. The HPC, for example, has been 100% linux in TOP500 for a few years already. Monopoly by FLOSS is still monopoly. Healthy competition is good for users and forces options to improve, see LLVM vs GCC.<p>To sum up: healthy FLOSS competition is welcome and needed.

&gt; FreeBSD&#x27;s network stack is (still) superior to Linux&#x27;s - and, often, so is its performance.<p>Where is this coming from exactly? The linked article about Facebook is 7 years old. The following benchmark shows the exact opposite: Linux&#x27;s network stack has long surpassed FreeBSD&#x27;s. And I would expect nothing else given the amount of work that has gone into Linux compared to FreeBSD.<p><a href="https:&#x2F;&#x2F;matteocroce.medium.com&#x2F;linux-and-freebsd-networking-cbadcdb15ddd" rel="nofollow">https:&#x2F;&#x2F;matteocroce.medium.com&#x2F;linux-and-freebsd-networking-...</a>

I ran FreeBSD servers for about a decade. Now all my servers are Linux with systemd. I&#x27;m liked FreeBSD then, I&#x27;m happy with systemd now. I have commits in both.<p>I&#x27;m glad there are some people who use and prefer FreeBSD and other init system now, because diversity in digital ecosystems is benefits the whole just as diversity in natural ecosystems do.<p>The shot taking at systemd here was disingenuous though. The author complained about the number of different systemd binaries and the lines of source code, but all these tools provide a highly consistent &quot;system layer&quot; with standardized conventions and high quality documentation-- it&#x27;s essentially the same argument made to support FreeBSD as a large body of kernel and userspace code that&#x27;s maintained in harmony.

It feels like the title is wrong. Instead of saying &quot;Linux is bad because I encountered X problem in production, which would have been prevented by BSD&quot; the author goes on to list why BSD is better in general outside his specific use case.<p>Nothing wrong with the comparison probably, but I got the impression the author just really wanted to do the migration and found some reasons to do so, without actually needing it. Nothing wrong with that as well. It&#x27;s just the expectations set by the title that are off

&gt;There is controversy about Docker not running on FreeBSD but I believe (like many others) that FreeBSD has a more powerful tool. Jails are older and more mature - and by far - than any containerization solution on Linux.<p>If FreeBSD jails and Solaris zones were equivalent to Linux containers, we&#x27;d have seen them take over the backend already. We haven&#x27;t. They&#x27;re really useful, they provided a degree of safety and peace of mind for multi-tenancy but they&#x27;re not granular enough for what&#x27;s done with $CONTAINER_RUNTIME these days.<p>Jérôme Petazzoni has an old talk where he touches upon container primitives and compared them to jails : <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=sK5i-N34im8" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=sK5i-N34im8</a>

I ran a FreeBSD ZFS NFS server for a cluster for quite a while. I loved it. It was simple and stable. The thing that led me away from FreeBSD (aside from IT not being happy with an &quot;alternative&quot; OS), was that I needed a clustered filesystem. We outgrew the stage where I was comfortable with a single node and where upgrading storage meant a new JBOD.<p>Are there any FreeBSD-centric answers to Ceph or Gluster or Lustre or BeeGFS?

I don&#x27;t have enough experience with FreeBSD (outside of FreeNAS seven years ago), but I&#x27;ve never had any success getting it to run on a laptop. Every time I&#x27;ve tried installing it on a laptop I get issues with either the WiFi card not working, issues with the 3D accelerator card not working at all, or the lid-close go to sleep functionality not working.<p>I&#x27;ve been using Linux since I was a teenager, so it&#x27;s not like I am a stranger to fixing driver issues, but it seemed like no amount of Googling was good enough for me fix these problems (googling is much harder when you don&#x27;t have functioning wifi). As a result I&#x27;ve always just stuck with Linux (or macOS semi-recently, which I suppose is kind of BSD?).

This articulates most of my frustrations with the Linux world.<p>Some of the distros are very good, but some of us who have work to do cringe at the thought of bringing up newer versions of an OS just to check all the things that&#x27;ve broken and changed needlessly.

“ The system is consistent - kernel and userland are created and managed by the same team”<p>Their first reason is really saying a lot but with few words. For one, there’s no systemd. The init system is maintained alongside the entire rest of the system which adds a lot of consistency. The documentation for FreeBSD is also almost always accurate and standard. Etc etc<p>I think you also largely don’t need a docker or etc in it since jails have been native to the OS for decades. I’d want to do some cross comparison first though before committing to that statement.<p>Shouldn’t be lost that the licensing is also much friendlier to business uses. There’s afaik no equivalent to rhel, for that matter. This goes both ways though as how would you hire a FreeBSD admin based on their resume without a rhce-like FreeBSD certification program?<p>Edit-I’ll posit that since FreeBSD is smaller an entity wishing to add features to the OS might face either less backlash or at least enjoy more visibility from the top developers of the OS. Linus, for instance, just has a larger list of entities vying for his attention on issues and commits.

To be fair all of these reasons come down to personal preference (sans the TCP performance claim). E.g. he prefers FreeBSD’s performance monitoring tools to Linux’s monitoring tools, or he prefers FreeBSD’s user land to Linux’s user land. That’s fine but it’s not very persuasive.

Things I actually care about: kernel that supports my hardware, ZFS for secure snapshotted data, scriptable tools to manage NICs and ppp and VPNs, a fast optimised C++ compiler, the latest versions of dynamic language runtimes, a shell, a text editor, a terminal multiplexer, a nerdy window manager and an evergreen browser.<p>On that playing field, the integrated nature of FreeBSD is nice but it’s an asterisk on top of the kernel rather than anything approaching what makes up a the system part of an <i>Operating System</i>. Almost everything else comes from a third party (and I’m fine with that.)<p>I haven’t used FreeBSD as a daily OS for over a decade though. What’s the new coolness?

&quot;Btrfs is great in its intentions but still not as stable as it should be after all these years of development.&quot; may have been true years ago, but doesn&#x27;t seem to be anymore.

The key point is not Linux vs FreeBSD. It is simply choice. You have a real choice. Do it this way or that - do it your way. I like both Linux and FreeBSD but I deploy them differently.<p>I slap Linux on my servers and desktops and I deploy FreeBSD via pfSense on firewalls.<p>Sometimes I do experiments and try out BSD on the desktop which hasn&#x27;t worked out yet for me but I live in hope because I adore *BSD as much as I do Linux.<p>If BSD is the way to get your servers to do what you want then lovely. Do it and remember you have choice.

I use freebsd for one project on node&#x2F;js and for ssh jumper boxes. I have also been admining linux boxes since 99. I have no hate for systemd - there just was a learning curve. I like a basic rc.conf set up that freebsd has. Everything can go in this one file for startups. Binary updates have been around for years so doing security updates are easy with no need to rebuild world or compile. You can use pkg for third party installs (binaries) although they don&#x27;t always follow the version in ports. Security wise kern_securelevel &#x2F; ugidfw for security. freebsd update also allows for easy updating in major os releases. ZFS on root just works on freebsd. PF &#x2F; ipfw to me makes much more sense than iptables (I haven&#x27;ted really moved to nftables).<p>When I compare to ubuntu which is the OS I use for linux mostly now: * kvm is superior to bhyve in every way * automating security updates via apt are better than a combination of freebsdupdate&#x2F;pkg updates. Plus the deb packages are made by Ubuntu and just work. ports&#x2F;pkgs are third party on freebsd * rebootless kernel updates exist for ubuntu * It is easier to find people familiar with linux right away<p>Really though the learning curve of freebsd &lt;-&gt; linux is not high.

The Wireguard debacle scared me off from FreeBSD. It seems they put too much trust in committers and don&#x27;t have a solid enough review process.

While I get the author&#x27;s reasoning, it makes me wonder at what scale, portability and level of automation and disposability all of this is done.<p>Even if an OS is &#x27;better&#x27;, a VM with a short lifetime will generally be &#x27;good enough&#x27; very quickly. If you add a very large ecosystem and lots of support (both open source and community as well as commercial support) and existing knowledge, FreeBSD doesn&#x27;t immediately come to mind as a great option.<p>If I were to go for an &#x27;appliance&#x27; style system, that&#x27;s where I would likely consider FreeBSD at some point, especially with ZFS snapshots and (for me) the reliably and fast BTX loader. Pumping out BSD images isn&#x27;t hard (great distro tools!) and complete system updates (due to the mentioned &quot;one team does the whole release&quot;) are a breeze as well. This is of course something we can do with systemd and things like debootstrap too, but from a OS-image-as-deployable perspective this will do just fine.

First off FreeBSD FTW. I use it everywhere over Linux now for the first time in 25 years and couldn’t be happier. My only wish is that BSD had a better non-CoW file system. Databases and Blockchains are already CoW so it does irk me slightly to use zfs for them. That being said, I’ve never had a problem because of it.

I have a tangential question on this part:<p>&gt; I sometimes experienced severe system slowdowns due to high I&#x2F;O, even if the data to be processed was not read&#x2F;write dependent. On FreeBSD this does not happen, and if something is blocking, it blocks THAT operation, not the rest of the system.<p>I’ve seen this for a long time in Windows, where any prolonged I&#x2F;O brings the entire system down to its knees. But it also seems to affect macOS (which is based on FreeBSD) as a system, though it’s not as bad as on Windows. Has Windows improved on this over the years? I’m unable to tell.

Looking forward to your next article when you move your stuff back to Linux

I used FreeBSD for many years (on servers) between 2001-2009. I also used it as a personal machine in the 90&#x27;s. We used it for stability, at which it did well.The real problem was that everything was moving to Linux. The Linux kernel and community kept up with with bleeding edge hardware or software. Stability of Linux continued to improve and most people stopped compiling custom kernels anyway. I used to compile most user-space software too, and almost never do now. That largely negated the FreeBSD benefits.

FreeBSD is a nicer, more logical Unix than Linux in general. Now as soon as you have a package or hardware that you want to use and it&#x27;s not supported by FreeBSD let us know how that goes.

I have been forced to work far too much with Citrix Netscaler virtual networking appliances and while I can see how it was probably a great product before Citrix purchased it the amount of bugs and regular security holes in it is insane. Especially for a damn networking appliance!<p>That being said it also forced me to use FreeBSD a lot more than I ever would have otherwise and I have a lot of respect for the OS itself. I would not use it everywhere but it has amazing latency which makes it obviously great for networking.

Bikeshedding at it&#x27;s finest!

It&#x27;s 2022 and if you still can&#x27;t see the good in systemd then it&#x27;s you choosing ignorance.<p>Related: <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=o_AIw9bGogo" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=o_AIw9bGogo</a> -- The tragedy of systemd.<p>Where Benno Rice (FreeBSD Committer &#x2F; FreeBSD Core member) explains the value of something like systemd.

Congratulation, i did the same ~5years ago, and cant be any happier, jails bhyve dtrace zfs&#x2F;ufs pf geom-compressed pkg&#x2F;ports etcetc...nearly every day i find some useful features, and when try them out...they work!!

FreeBSD is most likely slower than linux in most scenarios. ZFS is supported natively in Linux ( Ubuntu ), jail are terrible compared to Docker, since Docker is very popular there are a millions tools built arround it, it&#x27;s not just for sand boxing, it&#x27;s a part of a complet development process. Who cares about boot process on a server seriously?<p>&quot;FreeBSD&#x27;s network stack is (still) superior to Linux&#x27;s - and, often, so is its performance.&quot;<p>This is wrong, if it was the case most large compagnies would use BSD, atm they all use Linux, the only large compagny using BSD is Netflix because they added some tls offloading in kernel for their CDN which could have been done in Linux btw.<p>imo don&#x27;t use tech that is not widely used, you&#x27;re going to reinvent the wheele in a worse way because tool a.b.c is missing.

<a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=28584738" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=28584738</a> not Linux.

I could have sworn I read the same exact thing in 1997.

tl;dr - FreeBSD has ample nice features for their use case, and is considerably simpler. Linux has loads of unneeded (for their use case) features, and so many cooks in the kitchen that the ongoing cognitive load (to keep track of the features and complexity and changes) looks worse than the one-time load of switching over to FreeBSD.

Also remember that Darwin, the kernel used in macOS, is in part derived from FreeBSD.

Isnt bsd&#x27;s tcp stack single threaded?

TL;DR - for no compelling reasons.

All the &quot;advantages&quot; of FreeBSD are really just personal preferences, and little more. E.g., FreeBSD jails are not a replacement for containerization in any way. The FreeBSD network stack is better? I&#x27;ll bet you can talk to a Linux kernel expert who will explain why exactly the opposite is true. And things being &quot;simpler&quot; in *BSD? Simpler is not always better. SystemD may be somewhat over-engineered, but it&#x27;s also powerful as hell and can do things the old rc.X system couldn&#x27;t dream of doing.<p>There&#x27;s nothing wrong with switching to another OS, but implying it&#x27;s because the other OS is somehow empirically &quot;better&quot; is misguided.