Unity builds lurked into the Firefox Build System

Note that this is not referring to the Game Engine Unity. It's just referring to #including .cpp files.

With the advent of LTO, unity builds are mostly a band-aid for poor management of header files. The Linux kernel project was able to net a ~40% reduction in compilation CPU-time just by pruning the contents of some key header files [1].<p>It really boils down to two rules:<p>1. Don&#x27;t declare anything in header files that is only used in one compilation unit. Internal structs and functions should be declared and defined in source files, and internal linkage used wherever possible. gcc and clang&#x27;s -fvisibility=hidden is useful here.<p>2. The more frequently a header file is included (whether transitively or directly), the more it should be split up. If a &quot;common&quot; or &quot;utility&quot; header file is included in 10000 source files, then any struct, function, etc. that you add to that file will have to be parsed 10000 times by the compiler every time you build from scratch, even if only 10 source files actually use the struct&#x2F;function that you added. gcc and clang&#x27;s -H flag is useful here.<p>[1] <a href="https:&#x2F;&#x2F;lore.kernel.org&#x2F;lkml&#x2F;YdIfz+LMewetSaEB@gmail.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;lore.kernel.org&#x2F;lkml&#x2F;YdIfz+LMewetSaEB@gmail.com&#x2F;</a>

&gt; This generally leads to faster compilation time in part because it aggregates the cost of parsing the same headers over and over.<p>But this also reduces the opportunity to parallelize compilation across multiple files because they have been concatenated into fewer build units, and each unit now requires more memory to deal with the non-header parts. For some build systems and repositories, this actually increases build time.

I used Unity builds for my projects basically forever, at some point I discovered the practice had a name and some debates around it.<p>It is a simple thing to do, and the gains are substantial, faster and simpler, less maintenance, especially across different platforms.<p>For big projects I simply cut them into several libraries.<p>I&#x27;ve seen some incredulous reactions, mostly from young coders, and I know that makefiles <i>should</i> be faster, but in practice I never found that to be true.

&quot;Lurked into&quot;?<p>You can lurk but surely you can&#x27;t lurk <i>into</i> something?

This used to be mandatory for nvcc&#x2F;CUDA, if you had multiple source files (not just headers) you had to #include all of them in your main file. It made me very uncomfortable.

I’ve been out of C&#x2F;C++ development for a long time but seem to remember that precompiled headers were a thing back in the day. That approach didn’t have the name space issues pointed out here. Why are precompiled headers not used anymore?

Interesting. I&#x27;ve been aware of this technique for years because of the SQLite Amalgamation, but that was always sold as a way to simplify distribution and perhaps improve performance of the binary. I hadn&#x27;t considered it as a build speed optimization, though that seems somewhat obvious in hindsight.

Unity builds mean that you can no longer use internal linkage safely anymore, and that&#x27;s not something I like to give up. It forces the codebase to follow a certain style that I don&#x27;t like. Hopefully modules will give the advantage of unity builds without this downside.

Headers and C style macros are probably the most unfortunate aspects of C (and by extension, C++).<p>So many hacks in compilers to try to work around this. A shame there is no language level fix for this nonsense.<p>Really wish there could be a C++—- that would improve on C in areas like this, and avoid all the incredible nonsense of C++. And no, not Rust or Go.

We leverage many third party C++ libraries with complex templates, concepts, and constexpr expressions that seem to require lots of CPU to compile. We&#x27;ve found unity builds to be almost 3X faster, so we make it the default for both developer and CI jobs.<p>But we keep a separate CI job that checks the non-unity build, so developers have to add the right #include statements and can&#x27;t accidentally reference file-scoped functions from other files. While working on a given library or project, developers often disable the unity build for just that project to reduce incremental build times. It seems to offer the benefits of both approaches.<p>Precompiled headers don&#x27;t give nearly the same speedup. We&#x27;re excited for C++ modules of course, but we&#x27;re trying to temper any expectations that modules will improve build speed.

The compilation-unit-per-file model (and in fact the whole concept of linking) are a legacy incremental build solution for C which somehow metastasized into fundamental requirements of building software on current OSes. It is an atrocity and should be disavowed by all developers.

I always use unity builds for all my projects now. That combined with using tcc as compiler (for C code) makes builds really fast. Another nice feature of unity builds is that I don&#x27;t need to declare functions twice and keep the declarations synced. It&#x27;s also nice to only have one place to find information about a function; people often put comments in header files that you can miss if you go to the definition.<p>All of those things combined make C programming more enjoyable.

Why can’t static analyzers analyze the main cpp that #include-s the actual code? I don’t understand that point.<p>And what were the resulting affects on build times?

Lots of game studios use Unity builds like this. It saves a massive amount of time. Last I heard it also improves Incredibuild performance which is another popular tool for decreasing build timed.

Another benefit not mentioned is optimization. The compiler may be able to inline more function calls when function definitions and callers are in the same unified compilation unit.

Link is 404: <a href="https:&#x2F;&#x2F;web.archive.org&#x2F;web&#x2F;20230505055736&#x2F;https:&#x2F;&#x2F;serge-sans-paille.github.io&#x2F;pythran-stories&#x2F;how-unity-builds-lurked-into-the-firefox-build-system.html" rel="nofollow">https:&#x2F;&#x2F;web.archive.org&#x2F;web&#x2F;20230505055736&#x2F;https:&#x2F;&#x2F;serge-san...</a>

The post was renamed, the URL changed accordingly: <a href="https:&#x2F;&#x2F;serge-sans-paille.github.io&#x2F;pythran-stories&#x2F;how-unity-builds-crept-into-the-firefox-build-system.html" rel="nofollow">https:&#x2F;&#x2F;serge-sans-paille.github.io&#x2F;pythran-stories&#x2F;how-unit...</a><p>(HTTP 301 on the old URL would have been appreciated)

To avoid some of these issues, it can be helpful in a project to require that all files including header files must be compilable on their own. Doesn&#x27;t get rid of all the problems (you can still depend on transitive includes without explicitly including them) but enforces a minimal amount of code hygiene.

Tried Unity builds recently for ClickHouse, but without success: <a href="https:&#x2F;&#x2F;github.com&#x2F;ClickHouse&#x2F;ClickHouse&#x2F;pull&#x2F;18952#issuecomment-819901667">https:&#x2F;&#x2F;github.com&#x2F;ClickHouse&#x2F;ClickHouse&#x2F;pull&#x2F;18952#issuecom...</a>

It&#x27;s another reminder how &lt;expletives omitted&gt; C++ is, but for a long time nothing better existed.

Been using unity builds forever. The trick is to also have a standard build and try to compile it every week or so to catch anything that might have been missed, like a source file that is missing a header and wont compile alone because it got the header through the unity build ordering.

Compilation units are a relic of a time where computers only had a few KB of memory. At this point computers are fast enough and have enough memory to compile the whole thing in one go faster than whatever gains doing change detection and linking will have.