We’ll Never Know Whether Monorepos Are Better

From the arguments I&#x27;ve seen, the top highlights of single vs multi repo:<p>Single repo enables company-wide code sharing and refactoring with the least amount of friction. E.g. A programmer can modify <i>&quot;foobar(x,y,z)&quot;</i> to <i>&quot;foobar2(z,y,x)&quot;</i> and update <i>all</i> the corporate-wide client code that calls it. This ensures all teams are <i>always</i> using the same code instead of Team A on helperutils_v5.2.3 and Team B on helperutils_v6.1.0 which are incompatible with each other.<p>The biggest downside seems to be slow performance for syncing huge repos. This was one of the motivations for Microsofts GVFS for git. It enables <i>sparse</i> downloads.<p>If there are good arguments for multi-repos <i>that are not related to performance</i> which override the benefits of company-wide consistency of code, it would be good to discuss them. (I&#x27;m talking about companies like MS&#x2F;Google&#x2F;Facebook and not consulting firms with a different repo for each customer project.)

There is no <i>better</i>, there is only tradeoffs, and some people prefer one tradeoff and others prefer another tradeoff. That&#x27;s just how it is.<p>So before making any siginficant changes one should discuss what the desired tradeoff is, and then measure the continuous change towards that tradeoff.<p>For instance monorepo is a goal for people who want interacting tools to be build in their corresponding state without need to consult someone else. Can you do that? Check out the repo (for 2-5 hours depending on your code size) and then just hit &quot;make&quot; and &quot;make test&quot; and it runs through? If yes it&#x27;s a success, if no something went wrong in the process.<p>PS: There are actually build tools that are language independent, e.g. make, CMake or Maven. You can just use these.<p>PPS: I&#x27;m personally really happy to see that monorepo in itself doesn&#x27;t resolve the dependency tracking for internal tools. I still believe in the other way around. A good dependency tracking tool can totally replace monorepos.

&gt; Rolling our own dependency-management tooling took a small team of engineers (including me) more than a year of very hard work, and there were problems.<p>I am amazed what is done in large organisations sometimes. No one would ever sign off on me doing something like this. How do you convince the number crunchers that its a good thing to build things like this if there is no measurable outcome?<p>Not trying to be rude or devalue OPs work, but I haven&#x27;t worked in an organisation that would allow for this in a long while.

Note that there is a third option: both.<p>You can split up a git monorepo with splitsh[1], and&#x2F;or pull in &amp; out with git subrepo[2]<p>This gets you the best of both worlds and avoid most of the cons of both, although you do get a new set of cons, like keeping them in sync and ensure tooling works in both places. It&#x27;s not too bad, though. Make sure you bless one or the other as the source of truth and make the other basically read-only, and add a hook to automatically merge from one to the other.<p>1: <a href="https:&#x2F;&#x2F;github.com&#x2F;splitsh&#x2F;lite" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;splitsh&#x2F;lite</a> 2: <a href="https:&#x2F;&#x2F;github.com&#x2F;ingydotnet&#x2F;git-subrepo" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;ingydotnet&#x2F;git-subrepo</a>

I&#x27;ve recently inherited a nearly 100% python microservice-monolith monorepo project, whos tooling was written in-house by the original developers, and then subsequently re-implemented using the tooling&#x27;s plugin system within the project. And even then, the entire project is coupled together with some bash scripts to actually run it.<p>Fuck me it&#x27;s a nightmare wrapped in a shit show.<p>Due to the way the build tool generates the app, not a single &#x27;standard&#x27; method can be used. Any developer niceties you use can&#x27;t be (think interactive debugging, code-completion, hot-code-reloading).<p>I can see why it was done... sorta. Our FE team ripped out their component and standardised development back to their platform norms. But now we have integration issues between front and back, and the BE devs are now subject to a 60 second front-end build each time we change some code and reload the application.

There are technical pros and cons to both, but monorepos in my experience can &quot;fix&quot; or at least sidestep many human&#x2F;big organization issues:<p>- NIH, teams re-creating instead of re-using other teams work either due to not knowing it exists or because they feel they cannot control it.<p>- The gatekeeper: &quot;just ask Bob to give you access&quot;, repeat for the next two to four weeks. Once you get really big, source code access is even used as a political tool between teams.<p>- The internal customers: Team A works on a common framework, teams B, C, D all use it. Good luck getting anyone of them to take a new release more than once in a blue moon, and everyone will be on a different version too so you have to support them all. Fun times.

&gt; <i>There’s no way to measure productivity in software</i><p>This is clearly not true<p>Edit: It seems obvious to me that OP has a problem with his team, other than with being able to measure pros&#x2F;cons of his technologically choosen path

Interesting how the HN comments are also dividing between &quot;it&#x27;s obvious that monorepos are better&quot; and &quot;it&#x27;s obvious that multirepos are better&quot;.

Everybody will agree that big codebase need some sort of componentization so that understanding some part of the code doesn&#x27;t require understanding the entire codebase.<p>I like multiple repos because it encourages that: having pieces of code that do not depend on each others, and having more well defined API.<p>The speed gain of the tooling is just a side benefit.

While there are certainly ups and downs to both, I&#x27;d much rather have to deal with the downs of multirepo myself.

You can have your cake and eat it by mixing both. I&#x27;ve been reading through the Fuchsia [0] source, and they essentially have a monorepo setup that&#x27;s made up of multiple repos.<p>They wrote a tool called jiri [1], and track projects in a manifest [2] repo.<p>[0] <a href="https:&#x2F;&#x2F;fuchsia.googlesource.com&#x2F;?format=HTML" rel="nofollow">https:&#x2F;&#x2F;fuchsia.googlesource.com&#x2F;?format=HTML</a><p>[1] <a href="https:&#x2F;&#x2F;fuchsia.googlesource.com&#x2F;jiri&#x2F;" rel="nofollow">https:&#x2F;&#x2F;fuchsia.googlesource.com&#x2F;jiri&#x2F;</a><p>[2] <a href="https:&#x2F;&#x2F;fuchsia.googlesource.com&#x2F;manifest&#x2F;" rel="nofollow">https:&#x2F;&#x2F;fuchsia.googlesource.com&#x2F;manifest&#x2F;</a>

Cross-language artefact and dependency management. Tell us more about the tool.

My conclusion in the monorepo&#x2F;multirepo debate is that for large code bases, current tooling isn&#x27;t up to the task, and switching between monorepo and multirepo changes one set of tooling problems for another.<p>With custom tooling, you can overcome almost any of the differences between the two approaches, enough so that the two approaches can start to look rather similar. There are ways to do atomic refactors across large numbers of repos, with the right tooling, and there are ways to use semver for components inside a monorepo.

So, here is some more anecdata for the pile. We go with the multi-repo model at work, and the number of repos we &quot;use&quot; is over 1100 right now, and growing about a hundred repos a month.<p>How did we get here? An (imo) extreme dedication to &quot;microservice all the things&quot;, a CI system which intentionally only allows one build artifact per repo, and a release system that encourages the separation of parts of a service&#x27;s components to minimize red tape for changes.

OT, what is it with Medium sites pushing onto my history state? Getting out requires at least two, frequently more applications of &lt;Back&gt;.

It all depends on your needs and architecture.

Anecdata:<p>At my $DAY_JOB we went with a mostly-monorepo solution (1 main repo and 2 small auxiliary repos), with multiple dependent modules inside it -- after me insisting on it, while the other dev wanted a bazillion of small repos -- we&#x27;re talking about just a really small dev team (3 devs initially, more later) -- but we still agreed that 1) each subfolder of the monorepo should be as independent as possible and only depend on the minimum subset of other modules it really need (so multirepo-ish behavior in a sense), while at the same time 2) allowing to change things in multiple modules and quickly check if everything still works (we used symlinking between the modules&#x27; folders).<p>Result:<p>I was babysitting our build and tooling for <i>months</i> and fixing more and more exotic edge cases with incompatible transitive dependencies&#x2F;multiple branches&#x2F;versioning&#x2F;sumodules building order etc. (In hindsight, we threw way too much complexity and requirements on an opensource tooling we used than it was able to handle - but we didn&#x27;t acknowledge that upfront, because at the beginning it looked like it would work fine).<p>If we went with a multirepo instead, it would have been even worse: all the pains that I described would still be there, plus we&#x27;d go crazy with non-atomic refactorings and merges.<p>IMO starting a project as a multirepo is a madness: things are messy and change too often at the beginning and it&#x27;s not worth the overhead. Maybe it makes sense to start with a monorepo but structure things in a way to make them splittable in the future, when things stabilize more. But even that is <i>way</i> more costly than expected.

This article missed a great opportunity to enumerate the advantages and disadvantages of either model.

Same with:<p>* Vim vs emacs<p>* Tabs vs spaces<p>* Static vs dynamic<p>* Monolith vs microservices<p>* Functional vs imperative<p>And a long etc. We&#x27;re dogmatic folks.<p>I fall in the multirepo camp.

Who cares about that? Which is better, vi or emacs?

monorepo is very difficult to get right - and once you do, you are pretty much stuck with the language&#x2F;tool choices since the switch usually takes months&#x2F;years.<p>multirepo is a nightmare for changes spanning multiple repos, but the freedom of using the best tool for the job is much easier to achieve.<p>or so is my thinking about these from experience.

Again, this article and in general engineers discussing Monorepo vs Multirepo - seem to be discussing about splitting code without splitting dependency graphs, CI&#x2F;CD and independent delivery. So they had &quot;religious arguments&quot; (exactly as said in the article). Holistic Modularization and SOA (i.e. not just splitting dev&#x27;s job or qa&#x2F;ops&#x2F;architect&#x2F;manager&#x27;s job, but holistic) is not an easy thing in general.<p>Lots of FUD on this topic, and although its good that the author has bought this up, its just adding to even more FUD. I&#x27;d go as far as to say the title is clickbait. The end topologies and tooling for monoliths are different. Not necessarily one is better than other, they are <i>just different</i>. There is no magic pill, there are right places&#x2F;requirements and right people&#x2F;experience&#x2F;culture to use each of them.<p>Mono-repo:<p>* Imagine it like: Building a new Ubuntu OS release for every Firefox update. Then work backwards<p>* Mono-repo needs completely different toolchain like borg&#x2F;cmake, to find out the whole dependency graph, rebuild only changes and downstreams, avoiding a 2-hour build, avoiding 10gb release artifacts, scoped CI&#x2F;commit builds (i.e. triggering build for only one changed folder instead of triggering a million tests), independent releases, etc<p>* Some more tooling for large repository and LFS management, buildchain optimization, completely different build farm strategies to run tests for one build across many agents, etc<p>* Must have separate team with expertise to maintain mono-repo with 100 components, and knows everything that needs to be done to make it happen<p>* Making sure people don&#x27;t create a much worse dependency graph or shared messes, because its now easier to peek directly into every other module. Have you worked in companies where developers still have troubles with maven multi-module projects? Now imagine 10x of that<p>* Making sure services don&#x27;t get stuck on shared library dependencies. Should be able to use guice 4.0 for one service, 4.1 for another, jersey 1.x f&#x2F;a, jersey 2.x f&#x2F;a, etc etc. Otherwise it becomes an all-or-nothing change, and falls back to being a monolith where services can&#x27;t evolve without affecting others<p>* Does not mean its easy break compatibility and do continuous delivery (no, there is database changes, old clients, staggered rollout, rollbacks, etc. contracts must always be honored, no escaping that, a service has to be compatible with its own previous version for any sane continuous delivery process)<p>Multi-repo:<p>* Dependency graphs have to be split for continuous delivery<p>* Must not get into multi-repo if you don&#x27;t have true Continuous Delivery practices (not continuous integration&#x2F;deployment as tools, but Delivery as a mindset)<p>* Do not have &quot;Release management&quot; and &quot;Release managers&quot; with multi-repos, usage of trunk-based practices<p>* Don&#x27;t split just code but split whole dep graph<p>* Proper artifact management and sharing, because shared components are enforced to be &quot;diligently&quot; managed, not just thrown into a folder<p>* Plan for multiple types of deliverables: Services (runtime), static artifacts, libraries&#x2F;frameworks, contracts, data pipelines, etc<p>Not even scratching the surface of anything here. Its just an introduction to the introduction. If the team believed they would get this resolved, they are wrong, both are just beginning of two journeys. With the other direction, we would have got a different blog post today, that&#x27;s all.<p>In the end, if one has &gt;10 services, they WILL have challenges whether its mono-repo or multi-repo without having holistic modularization. They will need componentization of the service, metrics isolation, build process isolation, issues&#x2F;project management isolation, team isolation, etc etc. The repo is the last thing to worry about.<p>Edit: Formatting

Monorepo is harder to get right initially, and you have to think things through. But once you do, you&#x27;re set for it. I love that there is so little code duplication.