Adrian Colyer dug into this a little further on the morning paper:<p><a href="https://blog.acolyer.org/2019/04/18/keeping-master-green-at-scale/" rel="nofollow">https://blog.acolyer.org/2019/04/18/keeping-master-green-at-...</a><p>His analysis indicates that what uber does as part of its build pipeline is to break up the monorepo into "targets" and for each target create something like a merkle tree (which is basically what git uses to represent commits) and use that information to detect potential conflicts (for multiple commits that would change the same target).<p>what it sounds like to me is that they end up simulating multirepo to enable tests to run on a batch of most likely independent commits in their build system. For multirepo users this is explicit in that this comes for free :-)<p>which is super interesting to me as it seems to indicate that an optimizing CI/CD systems requires dealing with all the same issues whether it's mono- or multi- repo, and problems solved by your layout result in a different set of problems that need to be resolved in your build system.
"Based on all possible outcomes of pending changes, SubmitQueue constructs, and continuously updates a speculation graph that uses a probabilistic model, powered by logistic regression. The speculation graph allows SubmitQueue to select builds that are most likely to succeed, and speculatively execute them in parallel"<p>This is either brilliant or just something built for a promotion packet
We're building some similar tech at GitLab, though without the dependency analysis yet.<p>Merge Requests now combine the source and target branches before building, as an optimization: <a href="https://docs.gitlab.com/ee/ci/merge_request_pipelines/#combined-ref-pipelines-premium" rel="nofollow">https://docs.gitlab.com/ee/ci/merge_request_pipelines/#combi...</a><p>Next step is to add queueing (<a href="https://gitlab.com/gitlab-org/gitlab-ee/issues/9186" rel="nofollow">https://gitlab.com/gitlab-org/gitlab-ee/issues/9186</a>), then we're going to optimistically (and in parallel) run the subsequent pipelines in the queue: <a href="https://gitlab.com/gitlab-org/gitlab-ee/issues/11222" rel="nofollow">https://gitlab.com/gitlab-org/gitlab-ee/issues/11222</a>. At this point it may make sense to look at dependency analysis and more intelligent ordering, though we're seeing nice improvements based on tests so far, and there's something to be said for simplicity if it works.
There's a nice middle ground between this and a one-at-a-time submit queue: have a speculative batch running on the side. This gives nice speedups (approaching N times more commits, where N is the batch size) with minimal complexity.<p>One useful metric is the ratio between test time and the number of commits per day. If your tests run in a minute, you can test submissions one at a time and still have a thousand successful commits each day. If your tests take an hour, you can have at most 24 changes per day under a one-at-a-time scheme.<p>I worked on Kubernetes, where test runs can take more than an hour-- spinning up VMs to test things is expensive! The submit queue tests <i>both</i> the top of the queue and a batch of a few (up to 5) changes that can be merged without a git merge conflict. If either one passes, the changes are merged. Batch tests aren't cancelled if the top of the queue passes, so sometimes you'll merge both the top of the queue AND the batch, since they're compatible.<p>Here's some recent batches: <a href="https://prow.k8s.io/?repo=kubernetes%2Fkubernetes&type=batch" rel="nofollow">https://prow.k8s.io/?repo=kubernetes%2Fkubernetes&type=batch</a><p>And the code to pick batches: <a href="https://github.com/kubernetes/test-infra/blob/0d66b18ea7e8d3f216287ad06b11042c12bc6e48/prow/tide/tide.go#L759" rel="nofollow">https://github.com/kubernetes/test-infra/blob/0d66b18ea7e8d3...</a><p>Merges to the main repo peak at about 45 per day, largely depending on the volume of changes. The important thing is that the queue size remains small:
<a href="http://velodrome.k8s.io/dashboard/db/monitoring?orgId=1&panelId=10&fullscreen&from=now-7d&to=now" rel="nofollow">http://velodrome.k8s.io/dashboard/db/monitoring?orgId=1&pane...</a>
I am still trying to wrap my head around a giant monolithic repo model instead of breaking codes into multiple repos.<p>At Amazon, for example, they have multi repos setup. A single repo represents one package which has major version.The Amazon's build system builds packages and pulls dependencies from the artifact repository when needed. The build system is responsible for "what" to build vs "how" to build, which is left to the package setup (e.g. maven/ant).<p>I am currently trying to find a similar setup. I have looked as nix, bazel, buck and pants. Nix seems to offer something close. I am still trying to figure how to vendor npm packages and which artifact store is appropriate. And also if it is possible to have the nix builder to pull artifacts from a remote store.<p>Any pointer from the HN community is appreciated.<p>Here is what I would like to achieve:<p>1. Vendor all dependencies (npm packages, pip packages, etc) with ease.
2. Be able to pull artifact from a remote store (e.g. artifactory).
3. Be able to override package locally for my build purposes. For example, if I am working on a package A which depends on B, I should be able to build A from source and if needed to build B which A can later use for its own build.
4. Support multiple languages (TypeScript, JavaScript, Java, C, rust, and go).
5. Have each package own repository.
Quite a premise: "Giant monolithic source-code repositories are one of the fundamental pillars of the back end infrastructure in large and fast-paced software companies."
I think that what works for companies like Uber/Google/Facebook is not applicable to the rest of fortune 500 or all of the rest of the companies.<p>disclaimer: I am one of Datree.io founders. We provide a visibility and governance solution to R&D organizations on top of GitHub.<p>Here are some rules and enforcement around Security and Compliance which most of our companies use for multi-repo GitHub orgs.
1. Prevent users from adding outside collaborators to GitHub repos.
2. Enforce branch protection on all current repos and future created ones - prevent master branch deletion and force push.
3. Enforce pull request flow on default branch for all repos (including future created) - prevent direct commits to master without pull-request and checks.
4. Enforce Jira ticket integration - mention ticket number in pull request name / commit message.
5. Enforce proper Git user configuration.
6. Detect and prevent merging of secrets.
Having been at both Lyft and DoorDash where I've been an engineer responsible for unit test health, I decided to do a side project called Flaptastic (<a href="https://www.flaptastic.com/" rel="nofollow">https://www.flaptastic.com/</a>), a flaky unit test resolution system.<p>Flaptastic will make your CI/CD pipelines reliable by identifying which tests fail due to flaps (aka flakes) and then give you a "Disable" button to instantly skip any test which is immediately effective across all feature branches, pull requests, and deploy pipelines.<p>An on-premise version is in the works to allow you to run it onsite for the enterprise.