Why Google Stores Billions of Lines of Code in a Single Repository (2016)

I feel terrible for anyone who sees this and thinks, “ah! I should move to a monorepo!” I’ve seen it several times, and the thing they all seem to overlook is that Google has THOUSANDS of hours of effort put into the tooling for their monorepo. Slapping lots of projects into a single git repo without investing in tooling will not be a pleasant experience.

Is it just me, or are a lot of people here conflating source control management and dependency management? The two don&#x27;t have to be combined. For example, if you have Python Project X that depends on Python Project Y, you can either have them A) in different scm repos, with a requirements.txt link to a server that hosts the wheel artifact, B) have them in the same repo and refer to each other from source, or C) have them in the same repository, but still have Project X list its dependency of project Y in a requirements.txt file at a particular version. With the last option, you get the benefit of mono-repo tooling (easier search, versioning, etc) but you can control your own dependencies if you want.<p>edit: I do have one question though, does googles internal tool handle permissions on a granular basis?

Well, it&#x27;s not so uncommon. For instance, OpenBSD, NetBSD repos are sort of monolithic. And, believe it or not, there are some advantages. For instance, let&#x27;s take a look at OpenBSD 5.5 [0] release notes:<p>&gt; OpenBSD is year 2038 ready and will run well<p>&gt; beyond Tue Jan 19 03:14:07 2038 UTC<p>OpenBSD 5.5 was released on May 1, 2014. While Linux is still &quot;not quite there yet&quot; y2038-wise. y2038 is a very complex issue, while it may look simple - time_t and clock_t should be 64-bit. This requires changes both on the kernel -- new sys-calls interfaces [stat()], new structures layouts [struct stat], new sizeof()-s, etc. -- and the user space sides. This, basically, means ABI breakage: newer kernels will not be able to run older user space binaries. So how did OpenBSD handle that? The reason why y2038 problem looked so simple to OpenBSD was a &quot;monolithic repository&quot;. It&#x27;s a self-contained system, with the kernel and user space built together out of a single repository. OpenBSD folks changed both user space and kernel space in &quot;one shot&quot;.<p>IOW, a monolithic repository makes some things easier:<p>a) make a dramatic change to A<p>b) rebuild the world<p>c) see what&#x27;s broken, patch it<p>d) while there are regressions or build breakages, goto (b)<p>e) commit everything<p>[0] <a href="http:&#x2F;&#x2F;www.openbsd.org&#x2F;55.html?hn" rel="nofollow">http:&#x2F;&#x2F;www.openbsd.org&#x2F;55.html?hn</a><p>[UPDATE: fixed spelling errors... umm, some of them]<p>-ss

Managing dependencies and versions across repos is a pain. Refactoring across repos is quite hard when your code spreads across repos considering the tree of dependencies.<p>Unfortunately Git checkout all the code, including history, at once and it does not scale to big codebases.<p>The approach that Facebook chose with Mercurial seems a good compromise ( <a href="https:&#x2F;&#x2F;code.fb.com&#x2F;core-data&#x2F;scaling-mercurial-at-facebook&#x2F;" rel="nofollow">https:&#x2F;&#x2F;code.fb.com&#x2F;core-data&#x2F;scaling-mercurial-at-facebook&#x2F;</a> )

Maybe I&#x27;m not cool enough to understand this, but I don&#x27;t see the draw for monorepos. Imagine if you&#x27;re a tool owner, and you want to make a change that presents significant improvements for 99.9% of people, but causes significant problems for 0.1% of your users. In a versioned world, you can release your change as a new version, and allow your users to self-select if&#x2F;when&#x2F;how they want to migrate to the new version. But in a monorepo, you have to either trample over the 0.1%, or let the 0.1% hold everyone else hostage.<p>Conversely, imagine if you&#x27;re using some tools developed by a far off team within the company. Every time the tooling team decides to make a change, it will immediately and irrevocably propagate into your stack, whether you like it or not.<p>If you were at a startup and had a production critical project, would you hardcode specific versions for all your dependencies, and carefully test everything before moving to newer versions? Or would you just set everything to LATEST and hope that none of your dependencies decide to break you the next day? Working with a monorepo is essentially like the latter.

This is clearly detrimental to external projects such as Go packaging, since <i>their own</i> developers will never be looking at dependency problems in the same way as outside groups.<p>Monorepo also bugs me because there will <i>always</i> be some external package you need, and invariably it’s almost impossible to integrate due to years of colleagues making internal-only things assume everything imaginable about the structure and behavior of the monorepo. There will be problems not handled, etc. and it leads to a lot of NIH development because it’s almost easier in the end.<p>Also, it just feels <i>risky</i> from an engineering perspective: if your repository or tools have <i>any</i> upper limits, it seems like you will inevitably find them with a humongous repo. And that will be Break The Company Day because your <i>entire</i> process is essentially set up for monorepo and no one will have any idea how to work without it.

Having worked at different companies adopting both monorepo and the multiple repos approach, I find monorepo a better normalizer at scale in consolidating all &quot;software&quot; that runs the company.<p>Just like what many commenters here have mentioned, the monorepo approach is a forcing function on keeping compatibility issues at bay.<p>What you don&#x27;t want is to end up in a situation where teams reinvent their own wheels instead of building on top of existing code, and at scale, I think the multiple repo approach tends to breed such codebase smell. [1] I&#x27;m sure 8000 repos is living hell for most organizations.<p>[1] - <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=kb-m2fasdDY" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=kb-m2fasdDY</a>

One of my former managers had worked a long time at Google and was present for the advent of Google’s in-house tooling developed around their monorepo.<p>His account was that it was basically accidental, at first resulting from short term fire drills, and then creating a snowball effect where the momentum of keeping things in the Perforce monorepo and building tooling around it just happened to be the local optimum, and nobody was interested in slowing down or assessing a better way.<p>He personally thought working with the monorepo was horrible, and in the company where I worked with him, we had dozens of isolated project repos in Git, and used packaging to deploy dependencies. His view, at least, was that the development experience and reliability of this approach was vastly better than Google’s approach, which practically <i>required</i> hiring amazing candidates just to have a hope of a smooth development experience for everyone else.<p>I laugh cynically to myself about this any time I ever hear anyone comment as if Google’s monorepo or tooling are models of success. It was an accidental, path-dependent kludge on top of Perforce, and there is really no reason to believe it’s a good idea, certainly not the mere fact that Google uses this approach.

Google&#x27;s handling of their source code makes me wanna work there.<p>I don&#x27;t like distributed version control systems with hundreds of repositories spread out. It makes management more complicated. I understand this is a minority view, but that is my experience. It was easier to work in a single Perforce repository than hundreds of Git or Mercurial repos.

It should be noted that the monolithic model is somewhat encouraged by the client mapping system in Perforce, which was Google&#x27;s first version control system so it is unclear to me if this was deliberate or just a side effect of the best VCS of the time.<p>I also still have doubts around the value of a monorepo, in the article they claim it&#x27;s valuable because you get:<p>Unified versioning, one source of truth;<p>Extensive code sharing and reuse;<p>Simplified dependency management;<p>Atomic changes;<p>Large-scale refactoring;<p>Collaboration across teams;<p>Flexible team boundaries and code ownership; and<p>Code visibility and clear tree structure providing implicit team namespacing.<p>With the exception of the niceness of atomic changes for large scale refactoring, I don&#x27;t really see how the rest are better supported by throwing everything into one, rather than having a bunch of little repos and a little custom tooling to keep them in sync.

Previous thread:<p><a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=11991479" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=11991479</a>

&gt; Google&#x27;s monolithic software repository, which is used by 95% of its software developers worldwide, meets the definition of an ultra-large-scale4 system, providing evidence the single-source repository model can be scaled successfully<p>This 95% number is the most surprising part of the article. That implies that the sum of engineers working on Android + Chrome + ChromeOS + all the Google X stuff + long tail of smaller non-google3 projects (Chromecast, etc) constitute only 5% of their engineers. Is e.g. Android really that small?

My company has a 50m LOC Python codebase in a monorepo. It works really well, given the rate of change of thousands of developers globally. That is only possible because of the significant investment in devtools, testing and the deployment infrastructure.<p>Here is &quot;Python at Massive Scale&quot;, my talk about it at PyData London earlier this year:<p><a href="https:&#x2F;&#x2F;youtu.be&#x2F;ZYD9yyMh9Hk" rel="nofollow">https:&#x2F;&#x2F;youtu.be&#x2F;ZYD9yyMh9Hk</a>

We work with an monorepo since Septemeber 2017. I wrote about the migration:<p><a href="https:&#x2F;&#x2F;timkrueger.me&#x2F;a-maven-git-monorepo&#x2F;" rel="nofollow">https:&#x2F;&#x2F;timkrueger.me&#x2F;a-maven-git-monorepo&#x2F;</a><p>Our developers like it, because they can use &#x27;mkdir&#x27; to create a new component, search threw the complete codebase with &#x27;grep&#x27; and navigate with &#x27;cd&#x27;.

I wish more developers knew of the wonderful &quot;repo&quot; tool[0] developed by the Android devs which allows a monorepo _perspective_ of many git repositories. Breakdown of the repo tool and example manifest files <a href="http:&#x2F;&#x2F;blog.udinic.com&#x2F;2014&#x2F;05&#x2F;24&#x2F;aosp-part-1-get-the-code-using-the-manifest-and-repo&#x2F;" rel="nofollow">http:&#x2F;&#x2F;blog.udinic.com&#x2F;2014&#x2F;05&#x2F;24&#x2F;aosp-part-1-get-the-code-u...</a><p>[0] <a href="https:&#x2F;&#x2F;source.android.com&#x2F;setup&#x2F;develop&#x2F;repo" rel="nofollow">https:&#x2F;&#x2F;source.android.com&#x2F;setup&#x2F;develop&#x2F;repo</a>

&gt; includes approximately one billion files<p>...<p>&gt; including approximately two billion lines of code<p>_also_<p>&gt; in nine million unique source files<p>I should insert a joke about how well the system would do if each source file contained more than two lines of code.<p>But seriously, this summary could use some work.

It&#x27;s not just devops that you need to pull off a large monorepo; the other big thing is a strong testing culture. You have to be able to rely on unit tests from across the code base being a sufficient indicator of whether your commit is good. AND a presubmit process that can compute which parts of the monorepo get affected by your diff, and run tests against them automatically before committing your diff.<p>Google not only has the above but also has a strong pre-submission code review process which catches large classes of bugs in advance.

Here is the video (with Rachel Potvin), predating the article by some months: <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=W71BTkUbdqE" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=W71BTkUbdqE</a>

I&#x27;ve used monorepo for few small related projects and it worked just fine for me. Much easier to make related changes across several projects.

This is probably a stupid question, but I couldn&#x27;t find an answer. Does this mean Google keeps all of its different products in all their different languages and environments in one repo? So like, Android lives in the same repo as Gmail, which is the same repo as all the Waymo code and the Google search engine code as well? That seems insane to me.

Version controlled repositories are like business offices.<p>You can have your entire company in one location, or the entire company in separate locations. The <i>most</i> important thing is the logical rather than physical organization: team structure, executive leadership, inter-org dependencies, etc. You can achieve autonomy and good structure with or without separate locations.<p>A single location reduces barriers, but at some point multiple locations can solve physical and logistical challenges. General rule of thumb is to own and operate office space in a few locations as possible, but at some point you have to take drastic measures one way or another.<p>(Notice that Google had to invent their own proprietary version control system just for their monorepo. And not even Google <i>actually</i> uses a single repo as the source of truth: e.g. Chromium and Android.)

Im sure properly organized it&#x27;s okay, but from what I&#x27;ve seen it&#x27;s mediocre at best, especially with legacy&#x2F;technical debt it&#x27;s a huge mistake.<p>Start breaking that repo apart, because it probably isn&#x27;t very&#x2F;hopefully depending on the debt that exists.

Seems like Google uses its own custom Source Control &amp; tools - <a href="https:&#x2F;&#x2F;www.quora.com&#x2F;What-version-control-system-does-Google-use-and-why" rel="nofollow">https:&#x2F;&#x2F;www.quora.com&#x2F;What-version-control-system-does-Googl...</a>.

<a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Conway%27s_law" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Conway%27s_law</a><p>&gt; &quot;organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations.&quot;<p>Interestingly, in light of the above adage, this massive repo is organized (if that&#x27;s the word for it) like a bazaar or flea market. (Rather than like a phone book <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Yellow_pages" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Yellow_pages</a> )

&gt; Trunk-based development. ... is beneficial in part because it avoids the painful merges that often occur when it is time to reconcile long-lived branches. Development on branches is unusual and not well supported at Google, though branches are typically used for releases.<p>This sounds like the SVN model to me where branches are cumbersome and therefore they are very rare. After getting used to the Git branching model where branches are free and merges are painless, it would be very hard to go back to the old development model without branches.

How does CI work with a monorepo? Do you always have to run all the tests and build all the artifacts? Or are there nice ways to say &quot;just build this part of the repo&quot;?

I have slight experience with both monorepos and smaller repos and I think they can both work. The advantage of smaller repos is that it forces different components to expose well designed API&#x27;s. Bigger repos make sense for products and embedded software, smaller repos make sense for platforms build up of small services communicating on the internet.

Is this really relevant for anyone except for &quot;google scale&quot; companies? For most teams, managing 30-40 services backed by git repos isn&#x27;t a huge task and doesn&#x27;t cause many problems.<p>Is there mature tooling that helps teams manage this, or is this proprietary google magic tooling?

I don&#x27;t understand why gitlab&#x2F;github&#x2F;bitbucket don&#x27;t provide better tools for monorepo. This is a topic pretty trendy. But there is absolutely no tools helping with control access, good ci, ...

The giant monorepo works only if you&#x27;re using SVN, with Git it would be tremendous

Sorry, but as someone who has been in orgs that do both, mono repo is a mistake. Constant needs to pull unrelated changes before pushing, pipelines requiring to grab the whole repo for dependencies, etc. I understand the arguments for mono repo, but never think it&#x27;s nothing that outweighs the cons.

I love these articles. Is there a wiki or collection of detailed descriptions of large company tech practices that isn’t marketing blargh.<p>I read years ago about Google data ingest, locator process but neglected to bookmark so now can’t find the reference.

Should probably have a [2016] tag.

(2016)

A repo including configuration and data.<p>How about we stop considering google an engineering leader and just a search leader?

A repo with configuration, secrets and data?<p>Can we stop considering google an engineering leader and just a search algorithm leader?

I think monorepos make a lot of sense when you&#x27;re talking about millions of lines of code. I&#x27;m not at all sure they make sense when you&#x27;re talking about billions.

I don&#x27;t care about that. For me this is incomprehensible:<p>Why the eff does Google have billions of lines of code in their repo?<p>I hope they are not counting revisions (e.g., if a single 1 million project has 100 revisions, that&#x27;s 1 million, not 100 million).<p>I have heard that they do count generated code (so it&#x27;s not all handwritten code). In that case again, I have two things to say:<p>- that&#x27;s a bad metric. I could overnight generate a billion lines of code with each line a printf of number_to_word of numbers from 1 to a billion. They want to measure the size of the repo? They should tell us the gigabytes, terabytes etc. But when it&#x27;s lines of code, it&#x27;s cheezy and childish to blow up the measure by including lines of generated code.<p>- But more importantly, I hope the generated code is 90% or more of that repository. Because any less than that would mean that Google engineers have handwritten 100 million or more lines of code through out the lifetime of the company, in which case I have to ask: what bloated mess do you have on your hands? I thought you guys were the top engineers of the world.