The CD Pipeline Manifesto

I commend anyone who’s taking a hard look at our current CI&#x2F;CD practices. Good work! Succinctly stating the problems is easier said than done.<p>I believe <a href="https:&#x2F;&#x2F;dagger.io">https:&#x2F;&#x2F;dagger.io</a> checks all these manifesto boxes and more. At least that’s where I’m focusing my attention.

I&#x27;m thinking before we build a CI&#x2F;CI pipeline, make sure there is a Makefile. Why have they gone out of style?

&gt; Without types, it is difficult to compose pipelines together.<p>I would gladly hear this argument expanded. It&#x27;s really not obvious to me that that&#x27;s the case.

I&#x27;m guessing this is relevant: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=42267316">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=42267316</a> <i>Show HN: Glu – Deployment pipeline framework as code</i> - Nov, 2024 - 2 comments<p>And, tellingly, it seems they still haven&#x27;t provided a &quot;why not ${other tool}&quot; anywhere that I can readily spot

I have a hot take on this. I don’t care how you build and deploy as long as it’s reproducible and the whole process can be tracked in their metadata. I’d rather have a process validating CI&#x2F;CD stages and artifacts metadata in a central db than unifying pipelines that won’t get standardized due communication complexity. This way I can have a conversation on visibility rather than code edge cases.

I had a look at the example glu deployment pipeline and I’m decidedly unimpressed.<p>Admittedly most of my criticism is related to the choice of Go as an implementation language: more than 80% of the code volume is error handling boilerplate!<p>Before the lovers of Go start making the usual arguments consider that in a high-level pipeline script <i>every</i> step is expected to fail in novel and interesting ways! This isn’t “normal code” where fallible external I&#x2F;O interactions are few and far between, so error handling overhead is amortised over many lines of logic! Instead the code becomes <i>all</i> error handling with logic… in there… somewhere. Good luck even spotting it.<p>Second, I don’t see the benefit of glu (specifically) over established IaC systems such as Pulumi — which is polyglot and allows the use of languages that aren’t mostly repetitive error handling ceremony.<p>This seems like an internally developed tool that suits the purposes of a single org “thrown over the fence” in the hope that the open source community will contribute to their private tool.

&gt; Pipeline definitions are scattered across multiple tools—GitHub Actions, Jenkins, ArgoCD, Kubernetes—and environments. This fragmentation leads to confusion, configuration drift, and duplicated effort.<p>So are they talking about some sort of meta language compiling into multiple yaml configs for the different environments or a single separate CI tool that has plugins and integrates with GitHub&#x2F;gitlab&#x2F;etc?<p>I do agree with them about the need for a real programming language. I hate yaml in gitlabs config, it is very hard to interpret how it will be interpreted. Things were much easier when I was scripting Jenkins even though I didn&#x27;t know or like groovy then with gitlab

I don&#x27;t think this is reasonable if you have a cross platform (web, android, iOS, macOS, Windows, Linux, FreeBSD, ...) app, things won&#x27;t be that clean, that only works if whatever you do is very simple, otherwise there will be some patchwork to build and test across all platforms - there&#x27;s just no way to run all of them local in your single platform computer whatever that is. Honestly a lot of what is there is not that useful, I don&#x27;t need types in the pipeline when scripting python for build integration logic, there&#x27;s nothing that types brings that are a must in that case.

Is any of this reproducible? Not sure why that requirement has been quietly overlooked.<p>I&#x27;ve worked in this space for a long time and can&#x27;t make head or tail of what glue is.<p>A motivating examplt would be help which I might have missed?

Well written. Ive long held these values, and never could express them so concisely. Kudos!<p>I look forward to seeing some matrix eval of impl strategies against these values

FTA:<p>&gt; The Fix: Use a full modern programming language, with its existing testing frameworks and tooling.<p>I was reading the article and thinking myself &quot;a lot of this is fixed if the pipeline is just a Python script.&quot; And really, if I was to start building a new CI&#x2F;CD tool today the &quot;user facing&quot; portion would be a Python library that contains helper functions for interfacing with with the larger CI&#x2F;CD system. Not because I like Python (I&#x27;d rather Ruby) but because it is ubiquitous and completely sufficient for describing a CI&#x2F;CD pipeline.<p>I&#x27;m firmly of the opinion that once we start implementing &quot;the power of real code: loops, conditionals, runtime logic, standard libraries, and more&quot; in YAML then YAML was the wrong choice. I absolutely despise Ansible for the same reason and wish I could still write Chef cookbooks.

Best pipeline I’ve had the pleasure to design is AWS CodePipeline via AWS CDK. Ticks all boxes. Uses pure TypeScript code.

None of these are a problem anymore since the advent of Nix.