New make --shuffle mode

--shuffle shouldn't break anything with a correctly written makefile, but it's a nice debug tool for flaky makes. If --shuffle breaks things, there's a dependency problem. Nice.

This is great. Short of having Electric Make and Electric Insight it's the best thing you can get in open source and I think shuffling should really be the default to stop people from making horrendous parallel unsafe makefiles in the first place - so that the rest of us never have to fix them.

Oh, I like it; there do need to be more ways to tackle the whack-a-mole situation that can arise with -j. Glad there&#x27;s a seed option to make it deterministic.<p>I did this sort of thing brute force once with a script that used find on a build output to make a big list of all the build artifacts, and then did in effect foreach target in artifacts ; make clean ; make target. Took forever of course, but the set of failed targets is a list of every single target with a missing dependency.

I wish this existed a few years ago when I was regularly working on large Make projects. Excellent work.

Meanwhile my coworkers refuse to run googletest tests in random orders, because their tests all rely on being run in the &quot;right&quot; order and affect global data...

I don&#x27;t know a lot of makefiles, but based on what was written in the article, I was just trying to design a makefile which enforces goal ordering using the &quot;correct&quot; way, i.e dependencies. Such a makefile should naturally be unaffected by --shuffle and by parallel execution.<p>Do I miss something or is this impossible short of duplicating the goals?<p>E.g., suppose you have goals a, b and c, which are independent and you want to define another goal d which runs first a, then b then c.<p>My first idea was to define additional goals to capture the dependency structure, like this:<p>d: x2 c<p>x2: x1 b<p>x1: a<p>But then nothing is stopping the shuffler from running c, b, a, x1, x2, d.<p>The only solution I see would be to copy the shell commands of b and c into new goals:<p>d: xc<p>xc: xb; &lt;shell commands of c&gt;<p>xb: a; &lt;shell commands of b&gt;<p>Which seems really unsatisfying.<p>Is there a better way?

This is sort of a &quot;generative&quot; (aka property-based) approach to testing makefiles. So, I&#x27;d like to see a variation that can find a minimal failing subset of the makefile rules, as in QuickCheck:<p>&gt; In QuickCheck, assertions are written about logical properties that a function should fulfill. Then QuickCheck attempts to generate a test case that falsifies such assertions. Once such a test case is found, QuickCheck tries to reduce it to a minimal failing subset by removing or simplifying input data that are unneeded to make the test fail. -- <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;QuickCheck" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;QuickCheck</a>

Hijacking the thread, but a command line option I would really like to have in make is a way to force command echoing, including those prefixed by &#x27;@&#x27;. It would be the opposite of &#x27;-s&#x27;. If such a feature already exists, please tell me.<p>People drive me crazy with Makefiles that hide almost everything, and with no &quot;verbose&quot; option. Sure, it looks nicer when things go right, but when it doesn&#x27;t, which is the main reason why logs exist, then it is terrible. I always have to use ugly tricks to reveal what the Makefile author has hidden from me, like using the shell &quot;-x&quot; option, or removing all the &#x27;@&#x27; with a sed command, if would have been so much easier having a command line option.<p>Good thing we are mostly using cmake now, it has a working VERBOSE flag, but still, manual Makefiles still exist, and they are usually pretty terrible, and that would be a nice option to help debugging these terrible Makefiles (not unlike --shuffle).

Very neat.<p>I&#x27;d really like something like this for systemd units. On embedded systems you want to have confidence that any permitted ordering will result in a working system - as there&#x27;ll be no-one there to restart it if not.

Time to add GNU make to the list of randomizers: www.debigare.com&#x2F;randomizers&#x2F;

Any way to set a seed, to reproduce failures trivially? This has been really useful with pytest-randomly. Update: Should&#x27;ve read further, it supports --shuffle=SEED. Nice!

Cool! How do I get CMake with `Unix Makefiles` generator to use `--shuffle` flag? Then I can use this same feature to find dependency issues in my CMakeLists.txt files...

Great feature. And shuffle isn&#x27;t just useful for finding build issues, it can actually improve performance in some really annoying edge cases!<p>The basic idea is this: Let&#x27;s say you have a target being built and one of the dependencies takes a long time to compile; say, a large auto-generated file. If the time to compile this file is very dominant and outlandish, then you <i>really</i> want to do it as early as possible (while satisfying all dependent constraints), no matter what order the rules are in. This is so that it can overlap with other work as much as possible. If you instead pick it very late in the build process, it can significantly extend the total build time.<p>A concrete example is one I had at a past job. We used CMake on a reasonably-sized and actively developed C codebase. Make took about 5 minutes to do a clean build. Switching to Ninja reduced that to 2:30s, on a clean build. That was great. But then, I used Shake -- a build system that ships with a Ninja-compatible build tool, just called &quot;shake&quot; -- to build the same project. 1m30s total![1]<p>It took me a while to realize this is because shake by default randomizes build plans, both to find errors, <i>and</i> to increase performance in cases like this. By just shuffling the build order, you will get a &quot;smooth&quot; distribution of build times, whereas always deterministically sticking with 1 plan can result in a build plan with very very poor &quot;tail latencies&quot;. This was the exact example I mentioned. The project built about 10 executables, and 1 of them was a very large auto-generated tool, and compiling it took 80% of the total wall clock time of all other jobs combined. So shake would often build this tool <i>first</i> or at least very early on, just by chance -- while Ninja and Make would often build it <i>last</i>, every single time. A very big impact on tail latency.<p>So the TL;DR is -- randomize your build plans, knock out the missing dependencies, stabilize and ensure determinism where possible, and be happy!<p>[1] Shake was only better on the &quot;rebuild entirely from scratch&quot; case versus Ninja. When doing incremental rebuilds with small edits, I basically couldn&#x27;t find any (meaningful) difference in their performance.