Non-Recursive Make Considered Harmful: Build Systems at Scale (2016) [pdf]

One should start with, &quot;Recursive Make Considered Harmful&quot; [0] to know where this is coming from.<p>The best architected use of make is in the FreeBSD build system [1,2] If you want to experience &quot;a system&quot; please give FreeBSD a try.<p>[0] <a href="http:&#x2F;&#x2F;aegis.sourceforge.net&#x2F;auug97.pdf" rel="nofollow">http:&#x2F;&#x2F;aegis.sourceforge.net&#x2F;auug97.pdf</a><p>[1] <a href="https:&#x2F;&#x2F;www.freebsd.org&#x2F;doc&#x2F;handbook&#x2F;makeworld.html" rel="nofollow">https:&#x2F;&#x2F;www.freebsd.org&#x2F;doc&#x2F;handbook&#x2F;makeworld.html</a><p>[2] <a href="https:&#x2F;&#x2F;www.freebsd.org&#x2F;doc&#x2F;en&#x2F;books&#x2F;porters-handbook&#x2F;porting-samplem.html" rel="nofollow">https:&#x2F;&#x2F;www.freebsd.org&#x2F;doc&#x2F;en&#x2F;books&#x2F;porters-handbook&#x2F;portin...</a>

Another interesting paper is &quot;Build Systems à la Carte&quot; <a href="https:&#x2F;&#x2F;www.microsoft.com&#x2F;en-us&#x2F;research&#x2F;uploads&#x2F;prod&#x2F;2018&#x2F;03&#x2F;build-systems-5ab0f42d0f937.pdf" rel="nofollow">https:&#x2F;&#x2F;www.microsoft.com&#x2F;en-us&#x2F;research&#x2F;uploads&#x2F;prod&#x2F;2018&#x2F;0...</a> which explains characteristics of some build systems &quot;static vs dynamic dependencies; local vs cloud; deterministic vs non-deterministic build rules; support for early cutoff; self-tracking build systems; and the type of persistent build information. ... We show that we can instantiate our abstractions to describe the essence of a variety of different real-life build systems, including Make, Shake, Bazel, and Excel, each in a dozen lines of code or so&quot;

Is it really worth blaming the tool? Has reimplementing make for the 50th time really improved things?<p>The fact is, building software that requires 500 dependencies and 500 sub-steps and 500 configuration options is going to be complicated. It&#x27;s complicated in the same way that implementing an operating system is complicated. There&#x27;s no way around it. The complexity is there because it&#x27;s inherent in the problem.<p>But it doesn&#x27;t have to be. Instead of spending 300 hours implementing Shake, or Rake, or Bake, or Cake, or Jake, or Take, why not spend those hours cutting down the complexity at the source? Trim your dependencies. Stop putting so many sub-steps and configurations into your build systems. Cause it&#x27;s the build systems with the 500 dependencies and 500 the sub-steps and the 500 configurations option that are harmful; not the tools

Functional build systems are a great idea. The main problem I had implementing such a system was the file system. It&#x27;s tough to capture all inputs and outputs, even with the best of intentions.<p>The other problem is integration. You can&#x27;t expect all 3rd party projects to suddenly adopt your build system, so eventually you have to invoke `configure`, `make`, `cmake`, `pkg-config`, `xcode`, and so on. While you can satisfactorily capture most of these inputs and outputs, it&#x27;s non-trivial to do it completely, at some point you have something that works and it&#x27;s good enough.

<p><pre><code> While we have demonstrated that our approach works, we have not yet implemented all features of the build system, and hope to do so over the next few months </code></pre> This is a pretty major caveat. Almost damning in its significance, honestly. There are plenty of &quot;works, but haven&#x27;t quite implemented all of the old features&quot; projects littering the world. I love that there are learnings here, and those should be seen as the most important artifact of any project. I do wish there were paths to get those learnings back to the old systems, though. :(

<i>To validate our claims, we have completely re-implemented GHC’s build system, for the fifth and final time.</i><p>Famous last words.

Meh.<p>Linux and BSD build systems deal with most of these issues usually with wide support of a variety of recursive makes. Though RPM, DEB honestly suck and never really tried to solve issues automatically. Still drives me nuts that packages are tainted by the &#x27;gold&#x27; systems they are built on. The complexity of build systems means very few minds are up for it and most solutions are naive and end up with tons of patchy exceptions and work arounds.<p>ROCK Linux supported cross compiler capabilities and auto-detection of build parameters and dependency library tracking. (I was working on automated dependency ordering and QEMU based full cross builds, before I got a real job.) It was very robust and outside package developers breaking their own builds it worked solidly. No idea what cool things T2 Linux got up to after ROCK, but maintaining a fresh build system is hard. Build systems are always going to be fragile systems with complexity. The paper seems to be a survey of what they learned vs. definitely having any solution.

I&#x27;d like to issue a blanket ban on the &quot;considered harmful&quot; thing, as well as sharing print-optimized PDFs on the web for people on computers to read.<p>Both things are archaisms that can be easily avoided, and in the particular case of this article, the 2nd part of the title works just as well as the title.

Dropping in with my make horrorstory. I once had the pleasure of interacting with a 20k makefile (builds any one of 15 or so projects for any platform because &quot;splitting up the makefile would lead to code duplication since a lot of the makefiles would be similar&quot;) I&#x27;m told that makefile is over 40k lines today only a few years later.

Sorta reinventing the wheel?<p><a href="https:&#x2F;&#x2F;nixos.org&#x2F;nix&#x2F;" rel="nofollow">https:&#x2F;&#x2F;nixos.org&#x2F;nix&#x2F;</a>

My colleague @ejholmes wrote a cool tool that borrows heavily from `make` called `walk` (<a href="https:&#x2F;&#x2F;github.com&#x2F;ejholmes&#x2F;walk" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;ejholmes&#x2F;walk</a>). It&#x27;s written in Go and using a graph for dependencies so that tasks at the same horizontal level in the graph may be run at the same time. We use this at remind.com to significantly speed up our multi later AMI build times.

I&#x27;ve only used make a very little bit... but pretty much anyone can understand it for basic use.<p>This looks like you&#x27;re writing source code in another language that follows a kind of template, which you then compile and run, and then does stuff and is extremely complicated.<p>Seems like a failure to me? Shouldn&#x27;t something &quot;better&quot; be equally simple or simpler?

It is interesting that none of their examples of why make is considered harmful involved writing an idiomatic make file.<p>In fact, all of their examples of how bad make is involve the use of non-standard makefile generators, or non-standard extensions to make itself.<p>It is sort of like arguing that JavaScript is harmful, and then showing snippets of asm.js code to talk about how “tragic” the language syntax is.<p>Alternatively, I could argue that no one should use C because C++ template metaprogramming is too opaque, and, as further proof, my use of a non-standard preprocessor I implemented leads to 10,000’s of lines of deeply nested macro invocations.<p>(There are all sorts of problems with make, but I’m not convinced the authors actually understand them in enough detail to improve on it.)

This looks like a good stab at avoiding all the pitfalls of Make while still providing the same or better capability.

It looks great but most of make-like tools are not functionnal. So yeah functionnal everything is better but nothing is functionnal right now.