At the Mountains of Madness

I once had a robotic cat litter box that cleaned itself. Except once a week it would get clogged and I would have to spend a quality hour disassembling it, scrubbing off the feces embedded on the delicate parts, and reassembling it. Every two weeks when it heated up its artificial litter to dry, it would have missed a small piece of cat shit that when baked, filled my house with an aroma that I would not recommend you even try to imagine.<p>And of course it needed special, expensive supplies that you had to buy from the manufacturer because the bottles had numbered chips.<p>I eventually threw the damn thing out and now I just use a manual litter box. Takes 15 seconds a day to clean. It&#x27;s a chore but it&#x27;s a small predictable chore.<p>When I read about NixOS I remember that robot litter box. It seems like it solves a real problem of difficulty X but it brings five brand new problems from a parallel universe you didn&#x27;t know existed and they&#x27;re all written in an indecipherable language and have difficulty 10X.

Every time I look at NixOS, I think that it perfectly solves a problem that I only have once every 5 years, when buying a new computer. I think I even looked into it once to automate that exact process, but that idea fell apart at the first line of Nix syntax. I&#x27;ll stick with OSX and `brew bundle` I guess...<p>But then I read a piece like this and remember that some people do have to plumb the depths of C&#x2F;C++ linkers, and I&#x27;m glad I&#x27;m not one of them.<p>Great post! FWIW I always want to know the prompt text when seeing an AI-generated image, I wish there were a convention around that.

It seems like every article about nix goes on and on about DLL hell. I&#x27;ve been using Debian&#x2F;Ubuntu for 15+ years and never really experienced dependency hell. I guess maybe this is thanks to hard work by Debian maintainers and rarely needing to run a bleeding edge library, but also, why do we need to run bleeding edge versions of everything and then invent an incredibly complicated scheme to keep multiple copies of each library, most of which are completely compatible with each other?<p>And then when there&#x27;s a security problem, who goes and checks that every version of every dependency of every application has actually been patched and updated? Why would I want to roll a system back to an (definitely insecure) state of a few months ago?<p>What problem does Nix solve that SO numbers (properly used) doesn&#x27;t?<p>I have many of the same questions about Snap and even Docker.

&gt; No such file or directory<p>Anyone who’s run into this problem remembers it! (This isn’t a Nix problem—this is just the baffling errors you get because a.out exists, but one of the libraries it needs does not, and the error message doesn’t distinguish that case.)<p>Anyway, Nix.<p>Nix has the Nix way of building things. Nix doesn’t give you standard tools. It gives you wrappers around the standard tools that force you to do things a certain way. Part of that is futzing around with RPATH—because Nix stores everything in an unusual location. The user experience around this is awful, if you ever run into a case where Nix’s tooling doesn’t automatically do the right thing for you. It’s not just RPATH, but also other paths.<p>What’s the solution?<p>Honestly—I think it would make sense for Nix to have a “cross compilation” mode where you tell it to cross-compile for other Linuxes. You know, something like pkgsCross.x86_64-generic-linux. This comes with all the cross-compilation headaches, but you know what? You <i>are</i> cross-compiling.

Post author here. Feel free to ask me any questions about the piece of software that I most regret having had to write.

=======<i>Technical Summary</i>========<p>Here&#x27;s a problem with NixOS:<p>1. Suppose we have a `.&#x2F;nixos_binary_program_with_glibc-newer` compiled on a NixOS machine against bleeding edge `glibc-newer`.<p>2. `.&#x2F;nixos_binary_program_with_glibc-newer` will have `&#x2F;nix&#x2F;store&#x2F;glibc-newer&#x2F;linux-ld.so` path hardcoded into its ELF header which will be used when the program launches to find all of the program&#x27;s shared libraries, and so forth. (And this is a fact that `ldd` will obfuscate!).<p>3. When `.&#x2F;nixos_binary_program_with_glibc-newer` is distributed to machines which use `glibc-older` instead of `glibc-newer`, the hardcoded `linux-ld.so` from (2) will fail to be found, leading to a launch error.<p>4. (3) will also happen on machines which don&#x27;t use nix in the first place.<p>=======<i>Will&#x27;s Solution</i>========<p>1. Use `patchelf` to hardcode a standard FHS `ld-linux.so` location into `nixos_binary_program_with_glibc-newer`&#x27;s ELF header (using e.g. `&#x2F;lib64&#x2F;ld-linux-x86-64.so.2` as the path)<p>2. Use a metaloader to launch `nixos_binary_program_with_glibc-newer` with an augmented `RPATH` which has a bunch of different `&#x2F;nix&#x2F;store&#x2F;<i>glibc-newer</i>` paths, so that nix machines can find a suitable `ld-linux.so` to launch the program with.<p>This will make `nixos_binary_program_with_glibc-newer` work on <i>any</i> machine, including both non-nix machines <i>and</i> nix machines (which might be running older versions of glibc by default)!

I&#x27;m still confused why static linking isn&#x27;t a more common solution to versioning issues. Software developers normally have no problem using an order of magnitude more resources to solve organizational problems. Is there any technical advantage to dynamic linking other than smaller binaries and maybe slightly faster load times from disk?

Completely missed that Nix solves RPM dependency hell, which is a superset of the shared library hell.<p>Another problem not solved by NixOS and most other distros is conflating and mixing dependencies in a messy, fragile way rather than having a clear separation between the OS and add-ons that FreeBSD and others have. Congruent with this is proper configuration and lifecycle management.<p>I&#x27;m also wondering about the security of this RPATH approach, if it does or doesn&#x27;t introduce vulnerabilities.

I love what they’re going for, but I couldn’t help but react negatively at finding out that I had been hyped up for a post on some small technical topic for an OS I don’t know of. Maybe title it “At the Mountains of NIXos Madness”? But then again I’m just a grouch! Well written article regardless, from what I was able to get out of it

Calling binaries using ld-linux used to be a popular way to get around noexec on filesystems, since the libraries are usually in a place that is executable..

What is that abominable diffusion output doing at the top of an otherwise interesting article?

I loved this post, and patchelf is a real gem of a utility.

<i>tl;dr: we are open-sourcing an internal tool that solves a problem that we think many NixOS shops are likely to run into. The rest of this post is just the story of how we came to write this tool, which is totally a skippable story.</i><p>The tool happens to be called Madness, thus the Lovecraftian reference in this piece.<p><i>Madness enables you to easily run the same binary on NixOS and non-NixOS systems</i><p><a href="https:&#x2F;&#x2F;github.com&#x2F;antithesishq&#x2F;madness">https:&#x2F;&#x2F;github.com&#x2F;antithesishq&#x2F;madness</a>