The End of CI

&gt; How often have you or a coworker pushed an empty commit to &quot;re-trigger&quot; the CI?<p>Zero, because unless my dog configured the CI, I’d just click a button to re-run it on the same PR&#x2F;commit&#x2F;branch and not “make an empty commit”.<p>End to end testing of large systems with hundreds or thousands of man-years of code in them will be slow because there is a large surface area to test and testing in individual modules doesn’t give the same kind of end to end coverage that testing everything together does. So while pre-commit would be good, I doubt it will replace server CI for all but the smallest repositories.

I always have to be careful responding to things like this because I get sucked into the trap of thinking “we” means myself, and it just needs to mean “most of us” for whatever they’re talking about to be worrisome or even true.<p>I do worry that a lot of people are missing the first principles where CI is concerned, but a few of us still remember. Whether that’s enough is always a question.<p>&gt; First, CI rarely knows what has changed, so everything needs to be rebuilt and retested<p>CI is the impartial judge in your system. That is the ultimate goal and much of the value. Don’t believe <i>me</i> when I say that your check in broke something. Believe the robot. It does not think, it does not interpret errors (which is why intermittent failures poison the well). It just knows that whatever you did was not good.<p>We make it stupid simple because the more ways you try to interpret things, the higher the likelihood the system will either miss an error or report imaginary ones. And in this case I am very sorry Mr Lasso, but it is in fact the hope that kills. Optimism makes us try things, things others would give up on. That’s what makes a piece of software good. Better. But too much and you start ignoring very real problems and make optimism into a liability. I’ve seen it over and over again, take a build with intermittent failures and people will check in broken code and assume it’s the intermittent problem and not their change.<p>Ideally, when you start breaking builds you’ve found your limits. But the build has to break when I break the code, and it needs to do it repeatably, because if I can’t reproduce the error I risk telling myself stories, and foisting off debugging to other people, which is a big no no.

I fail to see the point being made in the article. CI done right is the best thing that ever happened to me in terms of collaborative productivity.<p>Of course there are many pitfalls. Like with any tool it becomes trivial to use it wrong. Integrating less than once per day and deferring all testing and linting to the CI process is an anti-pattern and the issue is not the CI process but how the team chooses to use it.<p>For example I worked in a team of 10 and we were doing multiple production deploys per day. This was made possible by a great CI workflow. Everyone was running tests precommit and focusing on keeping the pipeline green. Yes I&#x27;ve seen the opposite as well, but that usually is a symptom of issues with the team not the general concept of CI.<p>Not saying that there isn&#x27;t room for improvement but all the real-time collaboration features I used on low-code platforms feel like a step back.

It seems like a core argument is the pre-commit tests that runs as commit hooks on the developers computer.<p>I have worked in a place where they did that, and I think the cons heavily outweighed the pros. I can not push incomplete work to a remote, I can not separate feature development and chores (eg. linting) because I _need_ to fix stuff in order to commit an push, etc.<p>&gt; Continuous Integration is a model thats tightly tied to how we run version control.<p>I would say that a pre-commit testing world is much tighter. CI, as many know it, as a passive observer. When stuff fails, it will let you know. You can still merge.<p>One thing that would be nice, however, would be the ability to run the entire pipeline locally. For GitHub actions, it indeed seems like there are viable options to do that.

&gt; But all technology evolves, and git is already showing its weaknesses. Fork-and-pull isn&#x27;t the final form of development workflows.<p>I kind of get what this is saying, but technology evolution doesn&#x27;t have to mean completely replacing said technology with something else.<p>I think that&#x27;s one weird thing about the software field, whereby we keep moving to these shiny new things that we think are better than the tools of yesteryear, yet in the end there is only a marginal gain in productivity.<p>Fork and pull is an incredibly productive and powerful workflow. CI is incredibly, incredibly useful. If these things were not the case, then neither of these would be even discussed by this article. There is a reason for their success - and it&#x27;s not because GitHub is the most ubiquitous code hosting service out there. Git is _actually_ pretty great. CI is _actually_ very useful and has secured codebases for decades at this point.<p>So if one were to proclaim the &quot;End of CI&quot; I really need to see a viable alternative that addresses the same problems as CI and significantly improves upon it. An incremental improvement is not enough to shift and rewrite everything - there needs to be a significant jump in ability, productivity, security, or something else in order for me (and I imagine many others) to consider it.

What&#x27;s that old saying? Don&#x27;t remove a fence unless you really know why it&#x27;s there.<p>CI isn&#x27;t just a remote compute scheduler. Its also the place that contains tooling, tooling which repeated across a dev team creates n * jobs replicas of that tooling - which run on a local machine would be fairly wasteful. Its also centralized because CI is one of the biggest vectors an organization runs; it contains credentials for outside services, credentials for deployment (which, if you&#x27;re SOX-bound cannot exist on a dev machine), and if you&#x27;re cryptographically signing binaries and doing reproducible builds it&#x27;s got those keys too. n * developer laptops makes that vector a planet sized blot. A lot of the policies I imagine dealing with that would be limiting developer access to their machines, limiting tooling to only approved or internal tooling.<p>It&#x27;d be a nightmare.

Some of this is good, and some of it I feel makes some fundamental misunderstandings. The good: &quot;tight feedback loops are really important&quot;. Absolutely. Getting feedback about a change you (as a developer) are making is key, and the longer that takes the more time is spent waiting which is annoying and concentration breaking. This is, in my experience, usually tied to the implementation of tests (read: not bloated and useless) and not really a core tenant of CI.<p>However, things like:<p><pre><code> [...] git is already showing its weaknesses. Fork-and-pull isn&#x27;t the final form of development workflows. </code></pre> Show that the author is perhaps conflating GitHub with Git.

I disagree with the POV of the article. That said, I do work at Spacelift[0], a specialized CI&#x2F;CD for Infra as Code (so you can either take this with a grain of salt, or more trust, due to the increased exposure).<p>&gt; First, CI rarely knows what has changed, so everything needs to be rebuilt and retested.<p>Yes, this is an issue with some CI&#x2F;CD systems, an issue you can solve however. See our Push Policies[1] based on Open Policy Agent. Test-case caching is also sometimes available (available in i.e. Go).<p>&gt; Running tests pre-commit is one answer.<p>Running tests locally for a quick feedback loop - sure, that&#x27;s fairly mainstream though (something you can use our local preview[2] for as well in the case of IaC). Running tests locally before directly deploying to prod - that would be a compliance and security nightmare.<p>The author presents what looks to me like a very &quot;move fast, break things&quot; attitude, that doesn&#x27;t really work in a whole lot of cases.<p>If your CI&#x2F;CD is slowing you down, make it faster. Easier said than done, I know, but a lot of people don&#x27;t even think about optimizing their CI&#x2F;CD, which you often can do, by being smarter about what you run and&#x2F;or parallelizing extensively and&#x2F;or caching[3].<p>[0]:<a href="https:&#x2F;&#x2F;spacelift.io" rel="nofollow">https:&#x2F;&#x2F;spacelift.io</a><p>[1]:<a href="https:&#x2F;&#x2F;docs.spacelift.io&#x2F;concepts&#x2F;policy&#x2F;git-push-policy" rel="nofollow">https:&#x2F;&#x2F;docs.spacelift.io&#x2F;concepts&#x2F;policy&#x2F;git-push-policy</a><p>[2]:<a href="https:&#x2F;&#x2F;docs.spacelift.io&#x2F;concepts&#x2F;stack&#x2F;stack-settings#enable-local-preview" rel="nofollow">https:&#x2F;&#x2F;docs.spacelift.io&#x2F;concepts&#x2F;stack&#x2F;stack-settings#enab...</a><p>[3]:<a href="https:&#x2F;&#x2F;github.com&#x2F;actions&#x2F;cache" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;actions&#x2F;cache</a>

This take doesn’t make any sense. What is the author suggest as a replacement, hooking your IDE up to prod via FTP and editing PHP files directly on the live server? Sure I’ve done this, as a <i>solo</i> developer on smaller projects, but this doesn’t scale even to a team of two or three, and it goes without saying I’m not relying on tests in this scenario and I have basically no ability to roll back.<p>“CI is frustrating!!” I hear ya, but this article does nothing to clarify a viable alternative.

A poor title; I&#x27;d say &quot;The current CI is inadequate, we need to begin with a new CI&quot;.<p>Why is the current CI slow and resource-heavy?<p>The language(s) are inadequate: every function can affect another function or data; every function &#x2F; method may have arbitrary side effects. Even if not so, the language does not export the dependency graph.<p>The build system is inadequate: because in the source code everything potentially affects everything else, a lot more needs to be rebuild to be sure than would be strictly necessary for the logic of the change. Even if not so, the build system does not export the dependency graph and the scope of the actual changes.<p>The tests end up inadequate: even if they are written competently, they cannot trust any info about the scope of the changes, so they need to test way more than logically required to be sure, or, worse, they actually need to re-test everything. Also, due to the way other systems (databases, etc) are represented in the test setup, they are hard to run in parallel.<p>Microservices were invented partly as a way to fight this: they make <i>very</i> clear bounds between independent parts of the system, and strive to keep the scope of every part small. Their small scale makes the problems with the CI tolerable.<p>What we need is better tools, better static analysis and compartmentalization of the code, more narrow waists [1] in our systems that allow to isolate parts and test them separately.<p>[1]: <a href="https:&#x2F;&#x2F;www.swyx.io&#x2F;narrow-waists" rel="nofollow">https:&#x2F;&#x2F;www.swyx.io&#x2F;narrow-waists</a>

My ideal currently looks like:<p>- trunk-based workflow. Small commits. No feature branches as a rule to be occasionally broken<p>- unit tests move left - run pre-commit (not necessarily run in CI). Emphasis placed on them as a refactoring tool.<p>- a critical, small suite of e2e and integration tests that block the pipeline before publication (fast feedback)<p>- latest build publication being constantly redeployed to production, even if no changes have taken place to exercise the deployment routines<p>- a larger suite of e2e and integration tests being constantly run against production, to give feedback when something isn&#x27;t quite right, but it&#x27;s not a disaster (slow, non-blocking feedback).<p>In summary, emphasise getting code into production, minimise blocking tests to critical ones, test in production &amp; notify when features are broken.<p>Why?<p>- Engineers spend too much time in test environments that give the illusion of the real thing. They lose touch with production as the Dev cycle increases in circumference.<p>- Enabling tighter feedback cycles by accepting that some features are important and some are not helps put the cost of change into perspective for the entire product team.<p>- Engineers get used to working in and around production on a daily basis. Production operations and observation of systems (&amp; users) are emphasised - protective bubbles for writing code are minimised.<p>You&#x27;re not trying to maximise code output, you&#x27;re trying to maximise the velocity of safe change, and you do that by understanding the environment (production) through your intelligence (observability of systems and user behaviour), so that you can employ your force (changes, code, features) rapidly and effectively, whilst maintaining the ability to quickly deal with unexpected problems (defects) along the way.<p>Disclaimer: might not be possible for your specific theatre of war for any number of reasons.

I like CI tools that run locally on a developer machine and run end to end tests with all Microservices locally. The secret is integrating early and often.<p>That&#x27;s why I wrote mazzle and platform-up.<p>Mazzle is a run server that is configured as a Graphviz file (.dot) and defines an end to end environment. Here&#x27;s an example graph file of a pipeline:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;samsquire&#x2F;mazzle&#x2F;blob&#x2F;master&#x2F;docs&#x2F;architecture.tb.png" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;samsquire&#x2F;mazzle&#x2F;blob&#x2F;master&#x2F;docs&#x2F;archite...</a><p>It&#x27;s largely a prototype of an idea. It&#x27;s infrastructure and pipelines as code but incorporates every tool you use from terraform, chef, ansible, puppet, Kubernetes, packer and shell scripts. My example code spins up a Consul and Kubernetes cluster with hashicorp vault and a Debian repository server, configured SSH keys on every worker, bastion and Java application. And Prometheus exporter and grafana. I haven&#x27;t got around to adding ELK yet. But it didn&#x27;t take long to do all these things due to Mazzle meaning it&#x27;s very easy to test complicated changes together.<p><a href="https:&#x2F;&#x2F;devops-pipeline.com" rel="nofollow">https:&#x2F;&#x2F;devops-pipeline.com</a> - Mazzle<p>Platform up is a pattern for local development that tests your Microservices locally all together. You use vagrant-lxc and ansible together to deploy everything locally. So you can test your secret management locally and deployment process. If your ELK stack is ansible driven you can even run your ELK stack locally as I did on a client project. <a href="https:&#x2F;&#x2F;GitHub.com&#x2F;samsquire&#x2F;platform-up" rel="nofollow">https:&#x2F;&#x2F;GitHub.com&#x2F;samsquire&#x2F;platform-up</a>

First of all, it should be a fundamental principle of practical software engineering to be able to execute every test on the developer&#x27;s machine.<p>Software development that relies on &quot;special&quot; environments is prone to break down sooner rather than later. If you cannot execute your tests on a fresh machine after half an hour of setup or so, something is fundamentally broken in your toolset.<p>In turn, this requirement means that in a well-designed development environment your integration frequency is only limited by the product of integration points (e.g., your branch and &quot;develop&quot;, or your branch and n other ongoing branches) and the time it takes to run the tests.

&quot;Most&quot; and &quot;everybody&quot; on HN often means &quot;we at the few megacorps around&quot;.<p>Let me assume instead that most software projects happen in the long tail of small companies, low LOC numbers, low developer head counts (even one or zero - consultants working only when needed.) In that world I saw deployments run when tests pass on one PC, deployed from that PC. In the case of scripting languages even with a git pull on the server and a restart. That works surprisingly well until the team gets larger. Then customers often resist the extra costs and burden of CI. Then some of them cave in after something bad happens.

&gt; CI is a post-commit workflow. How often have you or a coworker pushed an empty commit to &quot;re-trigger&quot; the CI?<p>Build can be triggered manually in every CI system I know. Why would I push empty commits?

Maybe this will happen, but I really doubt it will come anytime soon, especially since the original tweet mentions that local, laptop-based testing is enough, which is laughingly ridiculous in terms of resources, time and coverage for anyone who&#x27;s worked on large, real-life, many-year, bloated projects<i>.<p></i>: which is pretty much every project that survived the POC phase

I feel as though this article comes from a place of very subjective personal experience, perhaps one which is painful. I&#x27;ve seen people avoid git because merge conflicts are hard to get their head around, afraid of the terminal because GUIs are what they&#x27;re used to. CI&#x2F;CD... CX, when done right, allows for automating a lot of work and allowing you to become much more efficient while documenting build, integration, and delivery workflows in code. If Mr Rickard doesn&#x27;t wish to see Jenkins in 2025 (three years away, so not exactly far in the future), how does he expect to be building &amp; deploying 100s of services?

I’m hopeful that <a href="https:&#x2F;&#x2F;dagger.io" rel="nofollow">https:&#x2F;&#x2F;dagger.io</a> will help CI “shift left”! Especially the ability&#x2F;easiness to run it locally, and the built in caching&#x2F;DAG

&quot;I, for one, hope that we aren&#x27;t using Jenkins in 2025&quot;<p>Or at least, if we are, it&#x27;s a far more robust and capable system that&#x27;s actually really designed for dealing with all the different things that people need to do while building and testing software.<p>But surely most IDEs these days make it simple to run all unit tests locally? My usual M.O. is to manually run the tests locally that I believe will demonstrate my changes are &quot;working&quot; and rely on the full suite of CI unit tests to check I haven&#x27;t broken anything else. What other options are there?

Fundamentally, you can&#x27;t trust what worked on a developer&#x27;s machine and you need a centralized system (the CI) to validate your build. So if the author is right, this will likely comes with the development taking place closer to the CI than the CI taking place closer to the development.

I suspect a lot of the author&#x27;s frustration comes from working inside a particular bubble for a long time and perhaps genuinely not realising that plenty of software developers <i>are</i> working in other ways already and always have been.<p>Not everything has to be a massively scalable cloud-native web app with a dependency tree 27 levels deep and DevOps turned up to 11. Not everyone has jumped on the merge-to-trunk-and-deploy-every-five-seconds process train.<p>Plenty of developers still work with modular systems in teams that each focus on a particular part of those systems. They follow simple feature-based development processes. They have local development environments that provide near-instant feedback on their own workstations.<p>A lot of the problems mentioned in the article simply don&#x27;t exist in that kind of environment. It would never even occur to those developers that they might need to push work in progress to trunk just to be able to test it properly or that their team might have significant merge conflicts so often during normal development that they&#x27;d need to change their entire process to reduce the frequency.<p>It&#x27;s like Scrum. Some people like it and some people don&#x27;t. But the people who don&#x27;t like the rigid time boxes and ceremonial meetings probably aren&#x27;t going to join organisations that have those things. Sometimes people in those organisations then start to think that their way is the only way because they&#x27;ve been doing it for so long and never seem to hire anyone with different ideas.

If there are 5 people framing a house, you can&#x27;t have all 5 of them doing 5 different things on the same section of roof at the same time. You have to put down the beams, joists, eaves, rafters, battens, purlins, collar ties, wind braces, sheeting, flashing, etc, one on top of the other. So to save time, you have people prep their work before they begin, schedule deliveries to arrive at exactly the right time, work in pairs, etc.<p>The point of CI is the same: to batch your work into stages and apply it just when it is needed. Merging your work incrementally ensures putting in your pieces doesn&#x27;t stop everybody else from putting in theirs or making them re-do stuff. So CI isn&#x27;t going away.<p>What we <i>should</i> do is make the process more seamless. For example, IDEs should not be editing local files, they should be committing directly to a VCS branch, which should be immediately rebuilding an app, which should immediately redeploy in a cloud environment dedicated to an engineer. We can make it as fast as local development (it&#x27;s not like your local computer is faster than a giant server in the cloud).<p>If you&#x27;re thinking <i>&quot;but what about offline work?&quot;</i>, we literally have 250Mbps satellite internet worldwide now. If you <i>really</i> can&#x27;t get a stable internet connection, you can keep a local server to do development in. But the local server software must be 100% identical to what&#x27;s in the cloud. Towards this end, we must build fully-integrated, fully-enclosed, fully-self-hosted CI environments. DVCS will seamlessly link the local and remote environments.

Pre commit tests assume everyone properly installs and runs them. Ci forces a standard onto everyone.<p>It&#x27;s two different worlds.

What is typically needed is an easily reproducible reference CI environment. That is often approximated with a centralized reference CI environment, that is not necessarily easy to reproduce, in which case it is only instantiated very few times.<p>Left or right of VCS should not be a big question given the VCS is supposed to be decentralized, but for that to happen the whole dev process (or at least a very substantial portion of it) should be day to day decentralized at well, or at least with a (well tested) ability to decentralize when needed, without friction.<p>If we go back to the more general &quot;configuration management&quot; concept (than merely source code version control), it is obvious the ideal situation is when the whole pipelines can be freely reproduced, in which case you can usefully do it before the integration on a reference branch&#x2F;repo (and redo it after just to be sure because who knows if the whole configuration has been correctly captured: you better detect discrepensies early too - also because in some workflow you won&#x27;t test exactly the same branch before merging to a reference as what is the result of the merge)

I&#x27;m not sure that CI is going to go away, especially so long as developers are building on machines that are not capable (sometimes, hardware constraints, sometimes differences in OSes) of running the exact software that is run in production. A CI server (or paas) makes doing builds safer for production, and saves the developer a lot of time.<p>I&#x27;ve become a huge proponent of where you can (sometimes you have to cross-compile, or the compute job is so big), using the same os to develop that you deploy in. This grossly simplifies tooling and testing and has the side effect of focusing developer mindshare (and development hours) away from simulating a foreign OS on their local machine. That time that would have been spent getting X working in Windows (and maintaining it) or fixing some deploy to container script because the new MacOS broke something can be spent, if there&#x27;s nothing else better, fixing bugs in the actual product.

I.. don&#x27;t get the gist of the article.<p>It seems to show that the author likes to use the git commit hooks to run linters and unit tests. Which is ... debatable, but the real question is: why does this should affect CI?<p>I understand that discovering a problem during the tests after the code landed on main branch is ... disconforting, at best. I also understand the frustation of not being able to commit your changes in a precise way because the git hook required you to also touch other stuff.<p>What I don&#x27;t know is why can&#x27;t we run the CI on a merge-request flow, on the beefy servers, without blocking the local commits, sharing feature branches any time we want, and be checked only &quot;when ready&quot;. FWIW I worked on a project set up this way, with a jenkins job starting each time a MR was made, where the branch is applied &quot;locally&quot; in the main branch and the tests run, and a (needed for actual merge) approvation obtained only if the tests passed.

I don’t like the hand-wavey “git is already showing it’s weakness”, then provides no examples.<p>This is a twitter tier take with no insight to back it up except “things will continue to get better and soon things will be different!” But doesn’t paint a picture of what different might be.<p>I mean, yeah things are going to change, duh

I cannot agree more. VCs are betting on companied like railway, render, netlify and vercel. If their vision becomes true there won&#x27;t be CI&#x2F;CD. An interesting thing happens if CI&#x2F;CD disappears from the process. The developer loop looks very different. Today we focus a lot on day 1 operations, building, provisioning, deployments. The focus will shift to the other side of the developer loop, I.e day 2 operations, incidents, chaos, etc. The number of incidents will increases which implies you need different kind of automation for day 2 which a asynchronous in nature, async to the developer loop. The automation we saw for CICD in past few years gets propagated to day 2 operations which in my opinion is still very nascent. Just some thoughts in the space.

Right, we should just edit and test directly in production.<p>Jokes aside, one of the reasons why we still need a CI is it proves we can do a full clean build. Optimizations that rely on what actually changed are good for local dev.<p>One way to truly get rid of CI is to get rid of the need to do a full clean build. If we structure our build to be constructed from a series of immutables input files (easy with version control) with a series of pure (as in no side effects) build steps and hash the output file by its input + builds steps, then the concept of a clean build is meaningless.<p>Then you could even allow regular dev machines to push the build assets to a centralized build cache. However I would still want an independent &quot;known good&quot; oracle to rebuild and check the hashes. One could call this oracle a CI...

&gt; CI is often triggered by creating a pull request to merge a branch into main.<p>&gt; Today, a developer would be crazy to suggest anything other than git and trunk-based development for a project.<p>My understanding is that trunk-based development is about shipping a feature via multiple merges of incremental changes that are often single commits; the main opposing model is feature-branch-based development, and it&#x27;s with the latter model that the terms &quot;pull request&quot; and even &quot;branch&quot; are associated. Is my description of the terminology accurate and is the author of this article using the terminology confusingly?

Like others here, I read this article and largely disagree with nearly all of the points made, and for the same reasons.<p>However, most of this is a result of the author assuming that everyone and every organization echos their experiences and point-of-view (which isn&#x27;t uncommon in online writings).<p>If the article were titled something along the lines of &quot;My views on the long-term viability of CI&quot; and the article written accordingly, then readers would be more willing to ponder alternatives to, or even a redefinition of CI in the viewpoint of the author knowing that it&#x27;s a personal opinion piece.

Did I miss the hacker news post where “git has already shown its weakness.” ? Someone help me out here.

OP has no idea what is possible in CI when you integrate intelligence about your compiler&#x2F;linker into the build executor and run it massively parallel. CI isn’t going anywhere bc parallel cloud execution will always build faster than your laptop. Less time wasted and less heat on your lap. BigCos are already doing this (some for over ten years) and running laps around garden variety CI. They are investing more in CI today than ever because the assumptions the OP made are incorrect.

I have never been a huge fan of CI. I&#x27;ve always considered it a potential &quot;Concrete Galosh&quot;[0], and, in my case, the fox is usually not worth the chase; but I don&#x27;t work in the type of environment that usually necessitates CI.<p><i>&gt; As I wrote in Your Integration Tests are Too Long, developers become reliant on CI and won&#x27;t run the tests until merge-time.</i><p>That&#x27;s a big issue. I think that testing is <i>very</i> important, and integration testing should be done from the git-go. Anything that discourages early integration testing, is a problem.<p>Recently, I was invited to submit a PR to a fairly sizable project (I was the original author, but have not had much to do with it for the last three years or so).<p>I declined, because, in order to make the PR, I would have had to set up a Docker container, Composer, Jenkins, xdebug, PHPUnit, etc., on my computer, in order to run the full integration tests (I won&#x27;t submit a PR without running the tests, as that&#x27;s just rude).<p>For someone that is a regular backend engineer, like most of the team working on the system, that&#x27;s no big deal. For me, it&#x27;s a fairly big deal (I write frontend native Swift stuff, and don&#x27;t have infrastructure on my machine for that kind of work).<p>That means that they will have to do without a fairly useful extension that I could have added.<p>[0] <a href="https:&#x2F;&#x2F;littlegreenviper.com&#x2F;miscellany&#x2F;concrete-galoshes&#x2F;" rel="nofollow">https:&#x2F;&#x2F;littlegreenviper.com&#x2F;miscellany&#x2F;concrete-galoshes&#x2F;</a>

&gt;First, CI rarely knows what has changed, so everything needs to be rebuilt and retested.<p>That&#x27;s a problem with your building and testing tools. Even without CI you would still need to build everything and test everything if you have no way to do it just for what changed.<p>&gt;How often have you or a coworker pushed an empty commit to &quot;re-trigger&quot; the CI<p>Most CI solutions have a button to trigger a new version without a new commit.<p>&gt;Running tests pre-commit is one answer<p>Even with CI developers are likely running at least a subset of the test suite while they are developing.<p>&gt;Yet, there are roadblocks today that need to be fixed, like tests that don&#x27;t fit on your laptop<p>Either develop on a server while using your laptop as just an editor or have a test runner on the server.<p>&gt;Things are shifting left<p>The problem with this is that at large companies the changes you&#x27;ve already tested will always be rebased onto a newer version of the codebane which it hasn&#x27;t been tested with. Who runs tests for this newly rebased version? CI. Also for things like come review you can not push it left onto the devs machine. You will want linters and tests to be run for come review.

&gt; Today, a developer would be crazy to suggest anything other than git and trunk-based development for a project<p>Funny, I thought it would be crazy to suggest trunk-based development when everybody is so enamoured with feature branches (trunk-based means that the whole team commits directly into the main branch - so pretty much the opposite of using branches with merge requests).<p>Another counterpoint is that CI makes at least a lot of sense for cross-platform development. E.g. when you&#x27;re working on Linux, you can&#x27;t trivially check your code locally on macOS and Windows, or even just on different compiler toolchains unless you install all those things on your local dev machine.

I find it funny how CI is not continuous and not integration.<p>It’s inherently a batch process, so not continuous.<p>It’s an automated build and check system. It also runs on merge commits but this “integration” part is really marginal to the concept as represented by e.g. GHA.

<i>developers become reliant on CI and won&#x27;t run the tests until merge-time</i><p>I think there&#x27;s a whole lot of truth in this particular statement. I&#x27;ve worked with lots of devs who complete a story, commit it, and then find the tests aren&#x27;t passing any more and have to do a little more work to fix what they broke. And then tery complain that either tests slow them down or that the estimate for the story was too low. Having a test suite that <i>can</i> be run locally, and teaching the team to use it regularly, even if it&#x27;s wholly optional rather than running on a hook, improves team velocity significantly.

Sorry to disappoint but I think Jenkins will be here far beyond 2025.<p>However on the topic of real-time integration that is somewhat already here for no-code systems that are already divorced from traditional version control.

The next evolution will be a shared build system.<p>If the builds done by the developer can be cached and shared with the CI, CI&#x27;s role is lessened to just a gate-keeper. Most of the time, the cache is already warmed up by the developer&#x27;s build and CI is a noop. Imagine a CI that take 5 seconds to finish.<p>For that to become reality, the build environment needs to be truly hermetic and reproducible, so that the cache can be trusted. Remote builders also help establish trust.

I can imagine that editing code in a fashion similar to Google docs collaboration is the way forward. Especially if tests run continuously, giving instant feedback.<p>This doesn&#x27;t mean that daily tests will be a thing of the past, but potential conflicts can be found sooner.<p>I don&#x27;t know how commits would work. I feel like they would become a hindrance to productivity, but having atomic changes spanning across multiple files seems like a must for proper rollback scenarios.

Continuous integration originally was shifted left of VCS. There was a shared integration workstation where people would pair program and sign off code as they go. CI shifted right and can shift left again too:<p>&gt; It can be very tempting to substitute automated build software for an integration computer.<p><a href="http:&#x2F;&#x2F;www.extremeprogramming.org&#x2F;rules&#x2F;dedicated.html" rel="nofollow">http:&#x2F;&#x2F;www.extremeprogramming.org&#x2F;rules&#x2F;dedicated.html</a>

The article posits an academic theory of CI that biases towards very narrow models of software development: Namely kubernetes, and namely orgs that have the budget to experiment in these types of models where developer trust is a nice to have, rather than regulatory compliance.<p>So as a thought experiment it’s neat, but knowing that most orgs will never need Kube and will never develop software like a FAANG, it just strikes me as clickbait given the title.

There have been times where I would not know how to locally test code or did not want to install a bunch of dependencies on my system. I don&#x27;t think precommit workflows will entirely replace CI, but it&#x27;s rise will surely ensure that commits and commit history will be higher quality. Testing locally used to require installing additional dependencies and testing suites but this could be simplified with containers.

I have the feeling there is a „we need to do x pre-commit“ season right now, opinions like „The end of CI“ seem to just pop up.<p>I‘m not convinced by pre-commit arguments. C in CI isn’t only for „continuous“, it also happens to introduce „centralised“ and „certain“ (as in, it’s going to run whether you like it or not).<p>Coupled with good git style there really is absolutely nothing that needs to change for teams of 1 and bigger.<p>Just my 2€

<a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=12622796" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=12622796</a><p>&gt; There is such a build system, but I can&#x27;t remember the name right now. It tracks system calls to see every file opened by the compiler to produce exact dependency graphs (assuming compiler is deterministic).<p>&gt; The downside is that it&#x27;s Linux only.

My ideal workflow is:<p>As I declare that I start development on a task, the branch is made automatically from current trunk and testing environment is set up with this branch deployed. As I commit and push that environment gets updated.<p>Last part of the development before handing it off to a tester is merging current trunk into my branch and resolving any issues. When the tester picks up this task to test then another merge from trunk to task branch is performed automatically and if there are conflicts task is automatically pushed back to development to resolve issues so the tester can immediately move onto testing another task. If there were no issues with pre-test merge the tester tests if the task was implemented satisfactorily. Then, if trunk didn&#x27;t change in the meantime task branch is merged back into the trunk by the tester. If the trunk was changed then the testing part repeats. The issue is that only one development task can be tested at a time without the need to recheck all currently tested when one of them is merged into a trunk.<p>In case of small tasks, tester might gain familiarity with the task to recheck it quickly after trunk was changed and merged into the task. In case of large dev task it should be split between few testers.<p>Order of merging tasks into the trunk should be determined by testing team to minimize recheck needed after the merges.<p>If anything goes wrong at any stage of testing the task is immediately kicked back to developer.<p>In this setup developers do no manual branching and merge only current trunk to their branches and can pretty much work independently (unless two pick very overlapping tasks), they can work on multiple tasks at a time and react quickly to tasks that come back from testing with minor issues. And testers coordinate testing and merging tasks back to trunk. Testers are also in close contact with stakeholders so they can demo new features for them on testing environments and know the priorities so they can work on merging higher priority tasks first. They can even participate in designing the new features or improvements as they know the current state of built software best and interact with it the most. Once the design is starting to crystalize deveolpers might be asked to provide input on feasibility and complexity of the feature implementation.<p>When I say merge it might be better to do a rebase. I&#x27;d have to actually work in that system to see what&#x27;s better.

It seems to me like the article is advocating for a solution to a problem that isn’t there?<p>&gt; But all technology evolves, and git is already showing its weaknesses.<p>Such as?

Hmm I&#x27;m unconvinced. It&#x27;s already possible to have developers run tests locally and not run tests in CI, which is what this sounds like.<p>Nobody does it because developers can&#x27;t be trusted to actually run tests.<p>Perhaps if there was some way to cryptographically prove that you&#x27;ve run tests? That doesn&#x27;t sound possible though.

In $current_place, we have about 250 hours (!) of tests that are run in CI per each commit. By splitting this across many executors, the CI can finish in about 40-60 minutes.<p>The idea of &quot;running the tests locally&quot; may fit some projects but in a complex system it&#x27;s definitely a no-go.

CI is a collaboration practice. It means to share your work with your team very often, to not develop little parts in silos for weeks or months and then put it all together at the very end.<p>I think the author means &quot;the end of build servers&quot;. You can do CI fine without build servers.

This blog post is riddled with issues. The key ones that come to mind are:<p>- it conflates a build system (like bazel) with CI execution platforms (like buildkite)<p>- it ignores the origins of CI - build everything from scratch so none of your workflow optimizations affect anything

Next.js has a marketing blurb about hot-reloading &quot;at Facebook scale&quot; on the website. I didn&#x27;t know what this could mean, but apparently someone has been thinking about it!<p>The build is neither working nor not working. In fact, it isn&#x27;t even built!

CI isn’t going anywhere, and is the best thing to happen to teams in terms of collaborative efficiency, even if you dislike the fact that you need to write tests. Write less, more important tests and write smaller, more composable scripts.

I wouldn&#x27;t trust my computer to be the pre-commit judge if my code is good or not. It looks nothing like the target environment and has plenty of modifications. I&#x27;ll trust CI to do that check for me.

The value for me wasin the last two paragraphs. In short: What&#x27;s next? For the same reason we&#x27;re not still using COBOL (sans legacy applications), something else has to be next in the progression.

I can&#x27;t take someone that&#x27;s still using Jenkins very seriously.

`git commit —-amend` will change the sha, even if you don’t change a single thing. I’d be upset if someone was pushing totally empty commits, wantonly polluting the history, just to trigger CI.

Here lies the problem &quot;developers become reliant on CI and won&#x27;t run the tests until merge time&quot; as always it is not technology&#x2F;approach problem but developers.

The first point is not true at all. GitLab CI has a feature when it can run jobs conditionally based on what changed. We did the same with Jenkins, it&#x27;s not hard.

That&#x27;s cute. Too bad it won&#x27;t work for use cases where the end product needs 100+ GB RAM to be run in its entirety and MUST be integration tested e2e.

The author could be right, but I&#x27;m skeptical. The model described is always-online, yet SCM like git is very deliberately distributed so you can work offline.

Is anyone doing collaborative cloud-based editing with the ability to run all tests at any time? That sounds pretty great, although you still need deployment gating.

Try renaming the title of this to &quot;The End of Testing&quot;, does it still make sense? I didn&#x27;t think so.

Thought experiment: taking real-time pre-commit workflows to the logical extreme, humans just become slaves to the machines, sitting at keyboards making near-instantaneous real-time changes to running systems in response to changing conditions and needs.

&gt; First, CI rarely knows what has changed<p>If you need this, you can have this already. As far as I&#x27;m aware, anyone running a large monorepo (Google etc.) is doing some type of diff-aware test pruning, simply because you can&#x27;t run all of the tests on every change.<p>And for us small guys, there are tools you can use to do change-aware tests, &quot;Test impact analysis&quot; is one keyword for this (there may be others, coverage.py calls it &quot;who tests what&quot;: <a href="https:&#x2F;&#x2F;nedbatchelder.com&#x2F;blog&#x2F;201810&#x2F;who_tests_what_is_here.html" rel="nofollow">https:&#x2F;&#x2F;nedbatchelder.com&#x2F;blog&#x2F;201810&#x2F;who_tests_what_is_here...</a>).<p>&gt; I, for one, hope that we aren&#x27;t using Jenkins in 2025.<p>I certainly hope not. I had moved on from Jenkins in 2015. But I don&#x27;t think there&#x27;s much reason to think the author&#x27;s &quot;run the tests locally&quot; speculation is particularly likely to supplant a centralized CI&#x2F;CD system. As a simple argument against, for supply-chain integrity reasons, the tests and builds need to be run in a common, public environment. You&#x27;re not going to push anything to production that has only been tested&#x2F;built on your own machine. This is one of the core tenets of the modern software philosophy that lets you move extremely fast (deploying many times per day) and (perhaps I&#x27;m showing a lack of imagination) it&#x27;s hard to see an alternative here.<p>Here&#x27;s a counter-prediction:<p>1) In the short term, the pendulum will continue to swing back towards thin-clients (Github Workspaces etc.) which means we&#x27;ll see more emphasis on cloud-based test runners per-dev-environment, and making these faster.<p>2) The push towards improved cycle-time (both for developers running tests on their WIP code, and for pre-merge tests on branches) will continue which will mean that the &quot;change-aware test runner&quot; tech will propagate down from high-complexity codebases so that every test runner is expected to offer conditional compilation&#x2F;testing.<p>3) And finally, a bigger&#x2F;more speculative one - the &quot;build&#x2F;test graph&quot; (the potentially-per-file set of tasks required to create your binary artifacts and test them; the thing Bazel computes, or that pytest computes) and the &quot;CI&#x2F;CD job graph&quot; (the thing that GitLab defines, which might include &quot;run the build&#x2F;test graph&quot; and &quot;deploy this artifact to $environment&quot;) will meet in the middle and become the same thing, so that you can invoke exactly the same graph processing logic locally (if you choose) as your CI&#x2F;CD server would invoke on any given commit&#x2F;tag. Earthly are working in this direction for example.<p>&gt; or smarter build systems that produce near-live feedback<p>You can currently set up your IDE to re-run the tests in real-time. If you add &quot;who tests what&quot; to that, you can have your IDE re-running just the tests related to your diff in real-time, which might actually be fast enough.<p>It&#x27;s not clear to me that the build system is the thing that needs to run this; it looks more like the IDE to me. But perhaps the author might agree with my general suggestion that we should make real-time IDE tests logically identical to the per-commit CI&#x2F;CD tests if possible.