Write Gitlab CI Pipelines in Python Code

Another step in the endless cycle of configuration vs code.<p>It&#x27;s not an accident that we are in a deep cycle of constrained configuration languages (yaml&#x2F;json&#x2F;etc etc). People chose to go there because before that we had a cycle of using programming languages and people hated it, for all sorts of good reasons.<p>Now I see we are on the way back into adding wrappers around the static config files, to turn them back into programming languages. This is happening all over the place - because guess what, it turns out, people hate static config files too, for all sorts of good reasons.<p>I am not sure if we will ever reach a compromise here or if we are just going to have to put up with endless change and churn because nobody is ever happy or at least willing to just settle for things that are &quot;ok&quot; but not &quot;perfect&quot;.

Several weeks ago I poked around Github&#x27;s CI workflow references. It seems that there&#x27;s a clear path for crafting a workflow that simply grabs the text of a each issue (as it&#x27;s submitted) and appends it to the documentation before closing the issue, perhaps marking it as &quot;expected behavior&quot;. In short, an evening of tinkering could automatically turn all of a project&#x27;s bugs into features for the foreseeable future.<p>This would represent a quantum leap in software quality assurance.

What I would really like to see is a CI system that lets me write a script in a language of my choice instead of defining a pipeline config file. That way I can run the pipeline locally, put breakpoints in, etc.<p>Nuke [1] gets close but there are still a lot of tasks that don&#x27;t have C# bindings, such as publishing build artifacts and uploading test results.<p>While I&#x27;m dreaming about my perfect CI, I&#x27;d also like the ability to download benchmark results from previous commits so that I can generate trend graphs and publish them in the build results.<p>To do this right the CI system would have to have an API using REST, GraphQL, gRPC or some such API format that generates clients in many languages. That way they don&#x27;t have to maintain bindings in every language.<p>[1] <a href="https:&#x2F;&#x2F;nuke.build&#x2F;" rel="nofollow">https:&#x2F;&#x2F;nuke.build&#x2F;</a>

It&#x27;s almost like we&#x27;ve turned yaml into some sort of primitive assembly file that must be compiled by a higher-order language. What was once created for human readability, has now turned into a machine generated and machine parsed format.

&gt; We started this project because of Gitlab CI yaml files growing over thousands of lines.<p>Is this typical? All of our Gitlab CI files are well under 100 lines. What sorts of things are these pipelines doing that require so much configuration?<p>Our CI steps are basically:<p>* Build<p>* Run some static analysis<p>* Test<p>* Publish build artifacts<p>With each step taking only a few lines. Most of the “heavy lifting” is managed by other tools like npm, or some scripts we have checked into the project, and our CI process just kicks off those steps.

I think if we&#x27;re going that way, it would be nice to have a single library that can generate configurations for multiple CI systems (GitLab CI, GitHub Actions, Jenkins, CircleCI, etc)

It seems like a lot of people don&#x27;t understand that having a DECLARATIVE language&#x2F;configuration&#x2F;whatever such a huge advantage it&#x27;s insane. It&#x27;s so easy to write, you can never make logical mistakes, avoid all kind of bugs, it&#x27;s just easy to learn and you can just write the desired result.<p>I never understood these project where somebody makes a very easy declarative thing and makes it imperative.<p>Programming in Python is order of magnitude &quot;harder&quot; and error prone than writing a simple YAML file.

I have made something similar for Tekton (Kubernetes) with Jsonnet, <a href="https:&#x2F;&#x2F;mustafaakin.dev&#x2F;posts&#x2F;2020-04-26-using-jsonnet-to-generate-dynamic-tekton-pipelines-in-kubernetes&#x2F;" rel="nofollow">https:&#x2F;&#x2F;mustafaakin.dev&#x2F;posts&#x2F;2020-04-26-using-jsonnet-to-ge...</a> but the eternal cycle of configuration vs code is always bothering me in every few years and make me question myself.

This is cool, but I&#x27;d rather my CI jobs are just k8s or nomad or systemd jobs. Then the code we deploy is the same as the code we use to build, and it doesn&#x27;t matter WHAT you do in build land, go nuts and do whatever you want.<p>CI systems will either grow into general purpose code runners or they will wither and die.

Reminds me of the original atlassian bamboo pipeline as code impl that had you write the pipeline as Java or Groovy, can’t remember specifically. They got dunked on for it not being a declarative file.<p>Now we’ve got a pipeline being defined by a declarative file being generated by code.

Gitlab can load its ci file from a remote web server. What I have done is generate the yaml programmatically and this gets returned from the web server. Less than ideal but it at least allows dynamic creation

&gt; “We use bamboo at my job, and it&#x27;s java that generates a yaml file that is then shoved into bamboo. It&#x27;s a completely shitshow.”<p>&gt; “What I have done is generate the yaml programmatically and this gets returned from the web server. Less than ideal but it at least allows dynamic creation”<p>They amount of polarization in this comment section is making me dizzy.