I've long wished to be just writing (ideally) Python to define and run CI/CD pipelines. YAML is simply hell, `if` keys in such data description languages are cruel jokes.<p>Round-trip times for GitHub Actions are too high: sometimes you're waiting for 10 minutes just to run into a dumb typo, empty string-evaluated variable or other mishap. There's zero IDE support for almost anything beyond getting the YAML syntax itself right.<p>We have containerization for write-once-run-anywhere and languages like Python for highly productive (without footguns as bash has them), imperative descriptions of what to do. The main downside I see is it getting messy and cowboy-ish. That's where frameworks can step in. If the Dagger SDK were widely adopted, it'd be as exchangeable and widely understood/supported as, say, GitHub Actions themselves.<p>We currently have quite inefficient GHA pipelines (repeated actions etc.) simply because the provided YAML possibilities aren't descriptive enough. (Are Turing-complete languages a bad choice for pipelines?)<p>What's unclear to me from the article and video is how this can replace e.g. GitHub Actions. Their integration with e.g. PR status checks and the like is a must, of course. Would Dagger just run on top of a `ubuntu-latest` GHA runner?
Hey, lead developer for the Python SDK here.<p>I have a few examples in <a href="https://github.com/helderco/dagger-examples" rel="nofollow">https://github.com/helderco/dagger-examples</a> and I plan to add more.<p>There's also reference documentation in <a href="https://dagger-io.readthedocs.io/" rel="nofollow">https://dagger-io.readthedocs.io/</a> so you can get a birds eye view on what's possible.
Hi everyone, Dagger co-founder here. Happy to answer any questions.<p>We also released an update to the Go SDK a few days ago: <a href="https://dagger.io/blog/go-sdk-0.4" rel="nofollow">https://dagger.io/blog/go-sdk-0.4</a>
Haven't heard of Dagger before, but using IaC as code to model your pipelines (including pipelines of infra that itself is modeled as IaC) is absolutely the right way to do things. And doing it in a real programming language rather than some limiting yaml schema is so much nicer. Places I've worked at in the past would refer to these as meta pipelines or self-mutating pipelines.<p>For this reason, I love using the AWS CDK. Being able to model things in TypeScript was so much nicer than the janky flow of always feeling lost in some Ruby/JSON/YAML IaC template monstrosity.<p>Curious how Dagger differentiates itself from AWS CDK or cdktf.
FYI one of Dagger’s developers runs the most excellent, and breath of fresh air really, Ship It podcast, which focuses on CI/CD and Infrastructure as Code (and always be improving!).<p><a href="https://shipit.show/" rel="nofollow">https://shipit.show/</a>
Hi guys, love the idea and the code examples look simple and understandable. Nicely done!<p>One of the reasons we use proprietary pipelines is the automatic 'service principal' login benefits that exist on e.g. Azure Devops, where the pipeline doesn't need to authenticate via secrets or tokens, and instead the running machine has the privileges to interact directly with Azure. (See <a href="https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/azure-cli?view=azure-devops" rel="nofollow">https://learn.microsoft.com/en-us/azure/devops/pipelines/tas...</a> particularly "addSpnToEnvironment" parameter). I'm sure other clouds have something similar.<p>Running the same pipeline locally, there are ways to synthetically inject this, but there's no ready support in your framework yet for this (as ideally you'd have an 'authentication' parameter that you can set the details for). Is something like this planned?
Curious if cuelang just ended up being too much of a hurdle for onboarding. I like it and have used it quite a bit but there's something about the syntax that makes it impenetrable for many.
So far I've been using Earthly for structuring builds, and it's been brilliant for the most part. Dagger was interesting but a no-go without Python, so curious to see how it stacks up.
Years ago, I had inherited a complicated C code base with some clever Makefile jujitsu. None of the recent maintainers understood it at its core. They just put a little lipstick here, and a little there to keep things running.<p>I sat down and extracted the compiler/link flags, and then wrote a Python script to do the build. The code was smaller, and built faster.<p>Every “build” engine evolves from being a simple recipe processor to the software equivalent of a 5 axis CnC mill. Some things should not succumb to one size fits all.
Even if this was a great, well-built tool, which I am sure it is...<p>Doesn't this introduce a whole dimension of extra stateful complexity compared to configuration YAML?
I had a look at Dagger, but I'm having a lot of success recently just building CI/CD pipelines with Nix. My `.gitlab-ci.yml` file is then just a thin wrapper of `nix build` commands, and pipelines can be run locally in the exact same environment as CI/CD.
This makes a lot of sense but would it prevent developers from using the default diagrams already built into most CICD websites. I assume migrating to this would make the CICD pipeline look like one step from gitlab, bitbucket, and githubs perspective.
This is an absolute god send for data engineers! We often get blocked full stop with CI/CD headaches and the alternative of “no CI/CD” is just unbearable at times. This needs to catch on quick !
Pipelines as code is the way to go, no matter whether it is CI/CD, data or ML. It’s unfortunate that a lot of pipeline tools use YAML. Glad to see projects like Dagger that use Python instead.<p>However it is not clear for me what is the benefit of using it instead of calling commands like docker, pytest and kubectl from Python with Plumbum or similar library. Add Fire and it is trivial to create a complete DevOps CLI for your app that can run locally or called from GitHub actions.
This is something I’d love to try on my personal projects.<p>I recently used act[0] to locally test my GitHub actions pipelines and worked okay, the fact that I could interact with the Dagger API via a Python SDK could be even more convenient, will definitely try!<p>[0]: <a href="https://github.com/nektos/act" rel="nofollow">https://github.com/nektos/act</a>
Shameless plug:<p>I'm working on a similar project, portable pipelines built with Nim :)<p>The idea behind this project is to:<p>1. Dev and Prod pipelines runs the same code, the only difference is where are executed. Allows easy troubleshooting and faster development.<p>2. Tests output by default on each script.<p>3. Decouples the code from the image/container. This allows it to be embedded into any CI stack.<p>4. Complex rules for retries specific scripts rather than the whole pipeline or even the step (stage)<p>A typical pipeline definition looks like this:<p><pre><code> ```yaml
kind: pipeline
version: 1
policy: strict
steps:
- hello
hello:
- script: echo "hello world"
expected_output: "hello world"
expected_return_code: 0
retries: 5
```
```bash
./takito --tasks tasks.yml --step hello
```</code></pre>
Here's the demo video linked in the blog post: <a href="https://www.youtube.com/watch?v=c0bLWmi2B-4" rel="nofollow">https://www.youtube.com/watch?v=c0bLWmi2B-4</a><p>It goes step by step through the getting started guide from the Dagger Python SDK docs
Confusing. I initially thought someone ported the Dagger DI framework to Python: <a href="https://dagger.dev/" rel="nofollow">https://dagger.dev/</a>
This looks awesome! I’ve built similar CI logic using git and docker apis for an autograder system. Looks like I can throw out some of this code… and replace it with dagger.
Debugging and maintaining code is to be avoided at all costs. You may prefer writing code as a developer, but it doesn't provide value to the people paying your paycheck, and it doesn't result in better outcomes than simple composeable actions.<p>Proprietary CI/CD systems are a waste of time. If it's more complicated than a shell script, you need to strip it down.