The Pain That Is GitHub Actions

Already see people saying GitLab is better: yes it is, but it also sucks in different ways.<p>After years of dealing with this (first Jenkins, then GitLab, then GitHub), my takeaway is:<p>* Write as much CI logic as possible in your own code. Does not really matter what you use (shell scripts, make, just, doit, mage, whatever) as long as it is proper, maintainable code.<p>* Invest time that your pipelines can run locally on a developer machine as well (as much as possible at least), otherwise testing&#x2F;debugging pipelines becomes a nightmare.<p>* Avoid YAML as much as possible, period.<p>* Don&#x27;t bind yourself to some fancy new VC-financed thing that will solve CI once and for all but needs to get monetized eventually (see: earthly, dagger, etc.)<p>* Always use your own runners, on-premise if possible

This is the joy of HN, for me, at least. I&#x27;m genuinely fascinated to read that both GitHub Actions and DevOps are (apparently) so universally hated. I&#x27;ve been using both for many years, with barely a hiccup, and I actually really enjoy and value what they do. It would never have dawned on me, outside this thread, to think that so many people dislike it. Nice to see a different perspective!<p>Are the Actions a little cumbersome to set up and test? Sure. Is it a little annoying to have to make somewhat-useless commits just to re-trigger an Action to see if it works? Absolutely. But once it works, I just set it and forget it. I&#x27;ve barely touched my workflows in ~4 years, outside of the Node version updates.<p>Otherwise, I&#x27;m very pleased with both. My needs must just be simple enough to not run into these more complicated issues, I guess?

There is one thing that I haven’t seen mentioned: worst possible feedback loop.<p>I’ve noticed this phenomenon few times already, and I think there’s nothing worse than having a 30-60s feedback loop. The one that keeps you glued to the screen but otherwise is completely nonproductive.<p>I tried for many moons to replicate GHA environment on local and it’s impossible in my context. So every change is like „push, wait for GH to pickup, act on some stupid typo or inconsistency, rinse, repeat”.<p>It’s like a slot machine „just one more time and it will run”, eating away focus and time.<p>It took me 25 minutes to get 5s build process. Naive build with GHA? 3 minutes, because dependencies et al. Ok, let’s add caching. 10 hours fly by.<p>The cost of failure and focus drop is enormous.

I worked at companies using Gitlab for a decade, and got familiar with runners.<p>Recently switched to a company using Github, and assumed I&#x27;d be blown away by their offering because of their size.<p>Well, I was, but not in the way I&#x27;d hoped. They&#x27;re absolutely awful in comparison, and I&#x27;m beyond confused how it got to that state.<p>If I were running a company and had to choose between the two, I&#x27;d pick Gitlab every time just because of Github actions.

GitHub Actions started off great as they were quickly iterating, but it very much seems that GitHub has taken its eye of the ball and the improvements have all but halted.<p>It&#x27;s really upsetting how little attention Actions is getting these days (&lt;<a href="https:&#x2F;&#x2F;github.com&#x2F;orgs&#x2F;community&#x2F;discussions&#x2F;categories&#x2F;actions?discussions_q=is%3Aopen+category%3AActions+sort%3Atop" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;orgs&#x2F;community&#x2F;discussions&#x2F;categories&#x2F;act...</a>&gt; tells the story -- the most popular issues have gone completely unanswered).<p>Sad to see Earthly halting development and Dagger jumping on the AI train :(. Hopefully we&#x27;ll get a proper alternative.<p>On a related note, if you&#x27;re considering <a href="https:&#x2F;&#x2F;www.blacksmith.sh&#x2F;">https:&#x2F;&#x2F;www.blacksmith.sh&#x2F;</a>, you really should consider <a href="https:&#x2F;&#x2F;depot.dev&#x2F;">https:&#x2F;&#x2F;depot.dev&#x2F;</a>. We evaluated both but went with Depot because the team is insanely smart and they&#x27;ve solved some pretty neat challenges. One of the cooler features is that their caching works with the default actions&#x2F;cache action. There&#x27;s absolutely no need to switch out popular third party actions in favor of patched ones.

&gt; Trivial mistakes (formatting, unused deps, lint issues) should be fixed automatically, not cause failures.<p>Do people really consider this best practice? I disagree. I absolutely don&#x27;t want CI touching my code. I don&#x27;t want to have to remember to rebase on top of whatever CI may or may not have done to my code. Not all linters are auto-fixable so anyway some of the time I would need to fix it from my laptop. If it&#x27;s a trivial check it should run as a pre-commit hook anyway. What&#x27;s next, CI should run an LLM to auto-fix failing test cases?<p>Do people actually prefer CI auto-fixing anything?

This was an interesting read and highlighted some of the author&#x27;s top-of-mind pain points and rough edges. However, in my experience, this is definitely not an exhaustive list, and there are actually many, many, many more.<p>Things like 10 GB cache limits in GitHub, concurrency limits based on runner type, the expensive price tag for larger GitHub runners, and that&#x27;s before you even get to the security ones.<p>Having been building Depot[0] for the past 2.5 years, I can say there are so many foot guns in GitHub Actions that you don&#x27;t realize until you start seeing how folks are bending YAML workflows to their will.<p>We&#x27;ve been quite surprised by the `container` job. Namely, folks want to try to use it to create a reproducible CI sandbox for their build to happen in. But it&#x27;s surprisingly difficult to work with. Permissions are wonky, Docker layer caching is slow and limited, and paths don&#x27;t quite work as you thought they did.<p>With Depot, we&#x27;ve been focusing on making GitHub Actions exponentially faster and removing as many of these rough edges as possible.<p>We started by making Docker image builds exponentially faster, but we have now brought that architecture and performance to our own GHA runners [1]. Building up and optimizing the compute and processes around the runner to make jobs extremely fast, like making caching 2-10x faster without having to replace or use any special cache actions of ours. Our Docker image builders are right next door on dedicated compute with fast caching, making the `container` job a lot better because we can build the image quickly, and then you can use that image right from our registry in your build job.<p>All in all, GHA is wildly popular. But, the sentiment around even it&#x27;s biggest fans is that it could be a lot better.<p>[0] <a href="https:&#x2F;&#x2F;depot.dev&#x2F;">https:&#x2F;&#x2F;depot.dev&#x2F;</a><p>[1] <a href="https:&#x2F;&#x2F;depot.dev&#x2F;products&#x2F;github-actions">https:&#x2F;&#x2F;depot.dev&#x2F;products&#x2F;github-actions</a>

&gt; A few days ago, someone compromised a popular GitHub Action. The response? &quot;Just pin your dependencies to a hash.&quot; Except as comments also pointed out, almost no one does.<p>I used GitHub actions when building a fin services app, so I <i>absolutely</i> used the hash to specify Action dependencies.<p>I agree that this should be the default, or even the required, way to pull in Action dependencies, but saying &quot;almost no one does&quot; is a pretty lame excuse when talking about your <i>own</i> risk. What other people do has no bearing on your options here.<p>Pin to hashes when pulling in Actions - it&#x27;s much, much safer

To make sure that you can test CI locally, the best way I&#x27;ve found so far is to make sure the checks can run with Nix, and then keep the CI config itself as simple as possible and just call Nix.<p>As for reducing boilerplate in the CI configs, GitHub Actions is a programming language with support for functions! It&#x27;s just that function calls can only appear in very limited places in the program (only inside `steps`), and to define a function, you have to create a Git repository. The function call syntax is also a bit unusual, it&#x27;s written with the `uses` keyword. So there is a lot of boilerplate that you can&#x27;t remove this way, though there are several other yaml eDSLs hidden in GitHub Actions that address some points of it. E.g. you can create loops with `matrix`, but again, not general-purpose loops, they can only appear in a very specific syntactic location.<p>To really duplicate stuff, rather than copy-pasting blocks of yaml, without using a mix of these special yaml eDSLs, in the past I&#x27;ve used Nix and Python to generate json. Now I&#x27;m using RCL for this (<a href="https:&#x2F;&#x2F;rcl-lang.org" rel="nofollow">https:&#x2F;&#x2F;rcl-lang.org</a>). All of them are general-purpose yaml deduplicators, where you can put loops or function calls <i>anywhere</i> you want.

Usually if you’re using it, it’s because you’re forced to.<p>In my experience, the best strategy is to minimize your use of it — call out to binaries or shell scripts and minimize your dependence on any of the GHA world. Makes it easier to test locally too.

I have used Travis, CircleCI, GitHub Actions, GitLab Pipelines, AWS CodeBuild&#x2F;CodeDeploy, Bazel, Drone, GoCD, and Jenkins. And I have used GitLab, GitHub, and Bitbucket for hosting VCS files. (I&#x27;m the guy who manages this crap for a living, so I have used it all extensively, from startups to enterprises)<p>GitHub Actions is the worst possible CI platform - except for all the others. Every single CI platform has weird limitations, missing features, gotchas, footguns, pain points. Every single one requires workarounds, leaves you tearing your hair out, banging the table trying to figure out how to do something that should be simple.<p>Of all of them I&#x27;ve tried, Drone is the platonic ideal of the best, simplest, most generally useful system. It is limited. But that limitation is usually easy to work around and doesn&#x27;t impose artificial constrictions. However, you won&#x27;t find nearly as many canned solutions or plugins as GitHub Marketplace, and the enterprise features are few.<p>GHA is great because of things like Dependabot, and the million canned Marketplace actions, and it&#x27;s all tightly integrated with GH&#x27;s features, so you don&#x27;t have to work hard to get anything advanced or specific to work. Tight integration can save you weeks to months of development time on a CI solution. I&#x27;ve literally seen teams throw out versioning of dependencies entirely because they weren&#x27;t updating their dependencies, because there&#x27;s no Dependabot orb for CircleCI. If they had just been on GHA using Dependabot it would have saved them <i>literal years</i> of headaches.<p>Jenkins is, ironically, both the most full-featured, and the absolute worst to configure&#x2F;maintain. Worst design, worst security, worst everything... except it does have a plugin for everything, and a UI for everything. I hate it with the fire of a million suns. But people won&#x27;t stop using it, partially because it&#x27;s so goddamn configurable, and they learned it years ago and won&#x27;t stop using it. If anyone wants to write a replacement, I&#x27;m happy to help (I even wrote a design doc!).

&gt; A few days ago, someone compromised a popular GitHub Action. The response? &quot;Just pin your dependencies to a hash.&quot; Except as comments also pointed out, almost no one does.<p>I&#x27;m surprised nobody has mentioned dependabot yet. It automates this, keeping action dependencies pinned by hash automatically whilst also bringing in stable upgrades.

We use a combination of AWS autoscaling and Nix to make our CI pipeline bearable.<p>For autoscaling we use terraform-aws-github-runner which will bring up ephemeral AWS machines if there are CI jobs queued on GitHub. Machines are then destroyed after 15 minutes of inactivity so they are always fresh and clean.<p>For defining build pipelines we use Nix. It is used both for building various components (C++, Go, JS, etc) as well as for running tests. This helps to make sure that any developer on the team can do exactly the same thing that the CI is doing. It also utilizes caching on an S3 bucket so components that don&#x27;t change between PRs don&#x27;t get rebuilt and re-tested.<p>It was a bit of a pain to set up (and occasionally a pain to maintain), but overall it&#x27;s worth it.

Interesting. I‘m also moving our CI to GitHub actions after years of using Jenkins with custom pipelines written in groovy etc. I checked out GitHub actions every now and then to feel if a move finally makes sense. I started with simple builds then tested adding our Jenkins macOS agents as self hosted runners. Just yesterday I wrote two actions to build and test a new .net project. I was able to run the whole thing with „act“ locally before running it on GitHub proper. I also played around and created a custom action in typescript (kicked off from the available predefined templates) to see how much work maintaining that means. All in all I‘m super happy and see no bigger issues. But here are some things that might be a reason: I split CI in build system logic which should and need to run locally and just stuff that GitHub needs to execute. At best that means describing what runs in parallel, and making specific connections. Any complicated logic needs to be abstracted away behind a a setup that is itself testable. I handle it the same for our build system components. We use gradle a lot and of a few custom plugins which encapsulate specific build &#x2F; automations. It’s like dividing your problem into many smaller pieces which are tested and developed in isolation.<p>Next to json I also used travisCI and appveyor for projects. And they all had the same (commit and pray) setup that ai hate. I wish if „act“ was a tool directly maintained by the GitHub folks though.<p><a href="https:&#x2F;&#x2F;github.com&#x2F;nektos&#x2F;act" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;nektos&#x2F;act</a>

There’s a lot of confident people in this thread saying CI is easy if you “just” make it dumb and keep all the logic in scripts that you farm out to.<p>My experience is this works for simple scripts but immediately falls apart when you start to do things like “don’t run the entire battery of integration tests against a readme change”, or “run two builds in parallel”, or “separate the test step from the build and parallelise it even if the build is serial”.<p>It’s easy to wrap make build and go about your life, but that’s no easier than just using the GitHub action to call go build or mvn build. T<p>he complexity comes in “pull that dependency from this place that is in a private repository on GitHub&#x2F;AWS because it’s 100x faster than doing it from its source”, and managing the credentials etc for all of that stuff. This is also where the “it differs from running locally” comes into it too, funnily enough.

I don&#x27;t get the obsession with YAML and making things declarative that really should not be declarative.<p>I&#x27;m so much happier on projects where I can use the non-declarative Jenkins pipelines instead of GH Actions or BB pipelines.<p>These YAML pipelines are bad enough on their own, but throw in a department that is gatekeeping them and use runners as powerful as my Raspberry Pi and you have a situation where a lot of developers just give up and run things locally instead of the CI.

Whatever happened to picking the right tool for the job ?<p>It looks like they have a very specific and unique build process which they really should handle with something more customizable like Jenkins. Instead they&#x27;re using something that&#x27;s really intended for quick and light deployments for intense dev ops setup.<p>I really like GitHub actions, but I&#x27;m only doing very simple things. Don&#x27;t call a fork bad because it&#x27;s not great when you&#x27;re eating soup

Whenever I get mad at GitHub Actions, I refer to it by it&#x27;s true name: VisualSourceSafe Actions. Because that&#x27;s what it is, and it shows. If you check out their Action Runner&#x27;s source code[1], you&#x27;ll find the VSS prefix all over, showing it&#x27;s lineage.<p>[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;actions&#x2F;runner&#x2F;blob&#x2F;6654f6b3ded8463331fb0c96a73b0435775c7c4e&#x2F;src&#x2F;Sdk&#x2F;Common&#x2F;Common&#x2F;VssException.cs#L14" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;actions&#x2F;runner&#x2F;blob&#x2F;6654f6b3ded8463331fb0...</a>

GHA is full of such obure behaviours. One I recently discovered is that one action can not trigger another:<p>If one action pushes a tag to the repo, `on:tag` does not trigger. The workaround apparently is to make the first action push the tag using a custom SSH key, which magically has the ability to trigger `on:tag`.

Genuine question: what&#x27;s the GitLab equivalent of GitHub Actions?<p>I&#x27;m using GitHub Actions to easily reuse some predefined job setup (like installing a certain Python version on Linux, macOS, Windows runners). For these tyoe of tasks, I find GitHub actions very useful and convenient. If you want to reuse predefined jobs, written by someone else, with GitLab CI&#x2F;CD, what can I use?

CI environments like Gitlab or Github are my nemesis. Another technology that everyone swears is absolute necessary but somehow makes everything more complicated. The provided environments in companies so far are hell 100% the time and managed by inexperienced personnel with zero or little programming experience.<p>* Barely reproducible because things like the settings of the server (environment variables are just one example) are not version controlled.<p>* Security is a joke.<p>* Programming in YAML or any other config format is almost always a mistake.<p>* Separate jobs often run in their own container, losing state like build caches and downloaded dependencies. Need to be brought back by adding remote caches again.<p>* Massive waste of resources because too many jobs install dependencies again and again or run even if not necessary. Getting the running conditions for each step right is a pain.<p>* The above points make everything slow as hell. Spawning jobs takes forever sometimes.<p>* Bonus points if everything is locked down and requires creating tickets.<p>* Costs for infra often keep expanding towards infinity.<p>We already have perfectly fine runners: the machines of the devs. Make your project testable and buildable by everyone locally. Keep it simple and avoid (brittle) dependencies. A build.sh&#x2F;test.sh&#x2F;release.sh (or in another programming language once it gets more complicated, see Bun.build, build.zig) and a simple docker-compose.yml that runs your DB, Pub-Sub or whatever. Works pretty well in languages like Go, Rust or TS (Bun). Having results in seconds even if you are offline or the company network&#x2F;servers have issues is a blessing for development.<p>There are still things like the mentioned heavy integration tests, merges to main and the release cycle where it makes sense to run it in such environments. I&#x27;m just not happy how this CI&#x2F;CD environments work and are used currently.

We&#x27;ve all been there:<p>$ git l * cbe9658 8 weeks ago rejschaap (HEAD -&gt; add-ci-cd) Update deploy.yml * 0d78a6e 8 weeks ago rejschaap Update deploy.yml * e223056 8 weeks ago rejschaap Update deploy.yml * 8e1e5ea 8 weeks ago rejschaap Update deploy.yml * 459b8ea 8 weeks ago rejschaap Update deploy.yml * a104e80 8 weeks ago rejschaap Update deploy.yml * 0e11d40 8 weeks ago rejschaap Update deploy.yml * 727c1d3 8 weeks ago rejschaap Create deploy.yml

I am completely switching my mental model of what a ci&#x2F;cd system should be at the moment: i use docker compose for absolutely everything possible. unit tests? runs as part of the container build. linear build dependent steps? multi stage docker biuld. DAG of build steps? dependencies in docker compose. This way every developer has the same system that ci&#x2F;cd uses locally. debugging the dev setup is the same as debugging the ci&#x2F;cd. The purpose of the actual ci&#x2F;cd is reduced to handling&#x2F;configuring triggers, handling env vars&#x2F;secrets and triggering the docker compose command with the proper selected docker context<p>This also reduces the lock in by orders of magnitude.

These are all real pains, author definitely has done a lot of work in Github Actions; respect. I&#x27;m sure these notes will save a lot of people a lot of frustration in the future, since Github Actions isn&#x27;t going away --- it&#x27;s too damn convenient.<p>I wonder why they chose to move back to Github Actions rather than evaluate something like Buildkite? At least they didn&#x27;t choose Cloud Build.

This is a fascinating read as someone who is currently working on migrating CI platforms. We deal with hundreds of internal repos where many CI platforms just don&#x27;t scale well. We are probably going to choose Github Actions because it looks like the simplest, least-BS option. The problems the author mentions would be the least of our concerns.<p>Totally agree with other comments saying to keep as much logic out of CI config as possible. Many CI features are too convoluted for their own good. Keep it really simple. You can use any CI platform and have a shit time if you don&#x27;t take the right approach.

I wonder if the complexity of fixing trivial code mistakes in CI is worth it compared to catching them in a pre-commit hook.

There&#x27;s a meta-problem here: GitHub Actions is one of those things that when you first encounter it, is presented as: &quot;we&#x27;ve got it all figured out and all goin on here, and anyone who is scratching their head must be dumb&quot;. This pattern, in my experience, shows up frequently in the software realm. Then, typically there follows some period where you try to do whatever you need to do by reading docs and copying what you see others do. Frustration and head scratching grows finally culminating in a process of &quot;Ok WTF are the core concepts of this thing, what were they thinking when they designed it, what is it really going???&quot;.<p>The article is what you end up finding after that stage has been gone through.<p>The conclusion of course is that whoever invented this stuff really wasn&#x27;t thinking clearly and certainly didn&#x27;t have the time to write decent documentation to explain what they were thinking. And now the whole world has to try to deal with their mess.<p>My theory as to how this ends up happening is that the people creating the thing began with some precursor thing as their model. They made the new thing as &quot;old thing, with a few issues fixed&quot;. Except they didn&#x27;t fully understand the concepts in that thing, and we never got to see that thing. You&#x27;ll see many projects that have this form: bun is &quot;yarn fixed&quot;. Yarn is &quot;npm fixed&quot;. And so on. None of these projects ever has to fully articulate their concepts.

I don&#x27;t see sourcehut [0] mentionned here. I tested github and gitlab CI, sourcehut is MILES ahead. I&#x27;ll drop two key features here : - any CI run successful or not, gives you back a ssh URI so you can log into the machine to inspect&#x2F;tweak&#x2F;tinker - CI files are <i>NOT</i> in the project&#x27;s repository. no need to wrangle with your git branches when working on CI anymore<p>[0] : <a href="https:&#x2F;&#x2F;man.sr.ht&#x2F;builds.sr.ht&#x2F;" rel="nofollow">https:&#x2F;&#x2F;man.sr.ht&#x2F;builds.sr.ht&#x2F;</a>

My team uses GitLab and most other teams are on Azure dev ops. They keep trying to get us to switch telling us how amazing pipelines are. Glad to know we are not missing anything.

Should have been zero permissions by default. The current model is a mess of global settings, workflow permissions, and job tokens that nobody understands.

GitHub Actions is just a way to build a remote single point of failure into your pipeline. I don&#x27;t get why people do this. If GitHub goes down, or has problems for whatever reasons it can interfere with your deliverables to customers. I learned early in my career not to trust 3rd parties as any part of any mission critical process. 3rd parties will always fail you, it&#x27;s just a matter of time.

I only use GitHub Actions for auto assigning issues (and I never merge pull-requests directly; I will always handle pull-requests manually). Here is the entire file:<p><pre><code> on: issues: types: - opened pull_request: types: - opened permissions: contents: read issues: write pull-requests: write jobs: default: runs-on: ubuntu-latest steps: - run: gh issue edit ${{ github.event.issue.number }} --add-assignee ${{ github.repository_owner }} env: GH_TOKEN: ${{ github.token }} GH_REPO: ${{ github.repository }} </code></pre> I set the permissions to only allow writing to issues and pull-requests (so that if gh is modified to do malicious things (or has a security flaw that allows it to do malicious things even if not intended), it cannot affect anything other than issues and pull-requests). As far as I can tell from the documentation, this is correct (although can do things other than add assignees, and it does not seem that it can be set more finely), but if I am wrong then you can tell me that I am wrong.<p>Documentation for GitHub Actions says, &quot;If you specify the access for any of these permissions, all of those that are not specified are set to none.&quot; The article says &quot;I do think a better &quot;default&quot; would be to start with no privileges and require the user to add whatever is needed&quot;, and it would seem that this is already the case if you explicitly add a &quot;permissions&quot; command into your GitHub Actions file. So, it would seem that the &quot;default permissions&quot; are only used if you do not add the &quot;permissions&quot; command, although maybe that is not what it means and the documentation is confusing; if so, then it should be corrected. Anyways, you can also change the default permission setting to restrictive or permissive (and probably ought to be restrictive by default).<p>Allowing to set finer permissions probably would also help.

Not sure if I&#x27;m the odd one out here. I thoroughly enjoy making the best of whatever the company wants to use. The flavour of CI&#x2F;CD can be a debate similar to programming languages

1st mistake: rebasing in CI.<p>Stop rebasing.<p>This should only happen if absolutely necessary to fix major merge mistakes.<p>Rebasing changes history and I&#x27;ve seen more problems prevented from removing it as a CI strategy.<p>Every CI strategy I&#x27;ve seen relying on rebasing had a better alternative in SDLC. You just need to level up your project management, period.

GHA feels like a discontinued product that people use so they can’t switch it off.

Azure DevOps is nearly identical, but with slightly different zoo of issues that are less well documented in public sources.<p>It also has the problem of not having a local dev runner for actions. The &quot;inner loop&quot; is <i>atrociously slow</i> and involves spamming your colleagues with &quot;build failed&quot; about a thousand times, whether you like it or not.<p>IMHO, a future DevOps runner system <i>must</i> be an open-source, local-first. Anything else is madness.<p>Right now we&#x27;re in the &quot;mainframe era&quot; of DevOps, where we edit text files in baroque formats with virtually no tooling assistance, &quot;submit&quot; that to a proprietary batch system on a remote server that puts it into a queue... then come back after our coffee to read through the log printout.<p>I should buy a dot matrix printer to really immerse myself into the paradigm.

We recently discover that if the last person to change cron __schedule__ of the workflow is removed from the organization, workflow fails with cryptic errors.<p>It turns out, the last person to change cron __schedule__ (not the workflow file in general) is an &#x27;actor&#x27; associated with this workflow. Very, very confusing implementation. Error messages are even more confusing - workflow runs are renamed as &quot;{Unknown event}&quot; and the message is &quot;Email is unverified&quot;.<p>Link to docs: <a href="https:&#x2F;&#x2F;docs.github.com&#x2F;en&#x2F;actions&#x2F;writing-workflows&#x2F;choosing-when-your-workflow-runs&#x2F;events-that-trigger-workflows" rel="nofollow">https:&#x2F;&#x2F;docs.github.com&#x2F;en&#x2F;actions&#x2F;writing-workflows&#x2F;choosin...</a>

After using Gitlab CI for years and setting up some pretty complex scenarios, when I switched over to GitHub I found the UX to be pretty rough. Seems very opaque and I find the documentation to be at best hard to navigate.<p>Maybe it was just the pain of switching but that was my initial impression.

GitHub actions provide me the best experience I ever encountered when dealing with ops.<p>I pair it with bash scripts where I find important to run outside GitHub facilitating test and maintenance.<p>Although, I still find need to run multiple times or iterate over the actual GitHub action run time, which is a bit slow to iterate but find it best to catch any issues. If the dev feedback loop fixed it save me a lot of precious time. I know there’s a third party to run locally but it’s not the same…<p>Thus, bash scripting is great due to portability.

As another commenter said, it&#x27;s good to write as much CI logic outside the yaml file as possible.<p>I take this a step further and approach CI with the mentality that I should be able to run all of my CI jobs locally with a decent interface (i.e. not by running 10 steps in a row), and <i>then</i> I use CI to <i>automate</i> my workflow (or scale it, as the case may be). But it always starts with being able to run a given task locally and then building CI on top of it, not building it in CI in the first place.

If you are annoyed by gitlab-runner deprecating run command that I used to run pipelines locally, there is <a href="https:&#x2F;&#x2F;github.com&#x2F;firecow&#x2F;gitlab-ci-local" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;firecow&#x2F;gitlab-ci-local</a> . But it also opened my eyes to benefita of having runner invariant pipelines -- pipelines written in solution agnostic way. Use bash, make, just, doit or whatever.<p>Nothing beats having a single script to bootstrap and run the whole pipeline e.g. `make ci`.

Used to use GH actions quite a bit. At my current company we set up RWX Mint (rwx.com&#x2F;mint) and haven&#x27;t looked back. (disclaimer: used to work at rwx but no longer affiliated)

At Our World In Data we ended up using Buildkite to run custom CI jobs, integrated with GitHub, but on cheap, massive Hetzner machines. I can really recommend the experience!

When GitHub actions are stopped GitHub just goes straight for the nuclear SIGKILL. No, asking nice first with a SIGTERM...<p>This means that for anything that needs to gracefully cancel, like for example terraform, it&#x27;s screwed.<p>Want to cancel a run? Maybe you&#x27;ve got a plan being generated for every commit on a branch, but you push an update. Should be ok for GitHub to stop the previous run and run the action for the updated code, right? WRONG! That&#x27;s a quick way to a broken state.

&gt; Why do I need a custom token? Because without it, the release completes, but doesn&#x27;t trigger our post-release workflow.<p>This is so frustrating. Having to inject a PAT into the workflow just so it will kick off another workflow is not only annoying but it just feels wrong. Also not lots of operations are tied to my user which I don&#x27;t like.<p>&gt; It doesn&#x27;t help that you can&#x27;t really try any of this locally (I know of [act](<a href="https:&#x2F;&#x2F;github.com&#x2F;nektos&#x2F;act" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;nektos&#x2F;act</a>) but it only supports a small subset of the things you&#x27;re trying to do in CI).<p>This is the biggest issue with GH Actions (and most CIs), testing your flows locally is hard if not impossible<p>All that said I think I prefer GH Actions over everything else I&#x27;ve used (Jenkins and GitLab), it just still has major shortcomings.<p>I highly recommend you use custom runners. The speed increase and cost savings are significant. I use WarpBuild [0] and have been very happy with them. I always look at alternatives when they are mentioned but I don&#x27;t think I&#x27;ve found another service that provides macOS runners.<p>[0] <a href="https:&#x2F;&#x2F;www.warpbuild.com">https:&#x2F;&#x2F;www.warpbuild.com</a>

If you have complex multi-step actions I recommend tools like nx (for frontend projects) or Bazel. It massively simplifies caching parts of your CI workflows and works locally too.<p>We have a very complicated build process in my current project, but our CI pipelines are actually just a couple of hundred of lines of GHA yaml. Most of which are boilerplate or doing stuff like posting PR comments. The actual logic is in NX configuration.

We recently had a developer -- while trying to debug container builds for a version upgrade for a PR on their local branch -- accidentally trigger a deployment of their local branch&#x27;s docker container to production (!) while messing around with Github action workflow files in their pull request (not main).<p>Outside of locking down edit access to the .github workflow yml files I&#x27;m not sure how vulnerabilities like this can be prevented.

I&#x27;m surprised the author doesn&#x27;t mention environment secrets, which I think currently are the only way to avoid that anyone with push access to any repo also gets full access to all secrets (by pushing a new workflow file and triggering it). This makes org and repo secrets practically useless for any team where only admins or maintainers should have access to secrets.

There definitely are a ton of issues with GitHub actions. To add to the OP&#x27;s list:<p>- Self-hosting on your aws&#x2F;gcp&#x2F;azure account can get a little tricky. `actions-runner-controller` is nice but runs your workflows within a docker container in k8s, which leads to complex handling for isolation, cost controls because of NAT etc.<p>- Multi-arch container builds require emulation and can be extremely slow by default.<p>- The cache limits are absurd.<p>- The macos runners are slow and overpriced (arguably, most of their runners are).<p>Over the last year, we spent a good amount of time solving many of these issues with WarpBuild[1]. Having unlimited cache sizes, remote multi-arch docker builders with automatic caching, and ability to self-host runners in your aws&#x2F;gcp&#x2F;azure account are valuable to minimize cost and optimize performance.<p>[1] <a href="https:&#x2F;&#x2F;warpbuild.com">https:&#x2F;&#x2F;warpbuild.com</a>

Also an Earthly casualty here. Now having to look at Dagger.

I&#x27;m surprised that they were already using earthly and decided not to continue using it inside github actions. That is my favorite pattern so that what github actions is doing is the same as what I&#x27;d do locally. No `act` necessary:<p><pre><code> - name: Install Earthly if: steps.check_changes.outputs.relevant_changes == &#x27;true&#x27; uses: earthly&#x2F;actions-setup@v1 with: version: v${{ env.EARTHLY }} - name: Run tests and generate coverage summary if: steps.check_changes.outputs.relevant_changes == &#x27;true&#x27; run: cd src&#x2F;webapp &amp;&amp; earthly --build-arg GO_VERSION=${{ env.GOLANG }} +coverage-summary</code></pre>

I&#x27;ve run into similar pain points with GitHub Actions in past roles but I still very much use them&#x2F;get value. One approach that&#x27;s helped us at Flox is indeed using Nix and we&#x27;ve now seen customers start leveraging that. Significant drop in “works on my machine” issues and more reliable CI pipelines, etc... It’s not a silver bullet, but it offers a solid technical foundation for tackling these challenges. On the Flox side we are very much on the integrate and improve rather than full replace for scenarios like this one &lt;3 Happy to answer any Nix items on this!

While I agree that GitHub Actions could be more &quot;secure and safe&quot; by default most gaps are easy enough to fill. With <a href="https:&#x2F;&#x2F;github.com&#x2F;marketplace&#x2F;actions&#x2F;check-actions" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;marketplace&#x2F;actions&#x2F;check-actions</a> you can ensure timeouts, permissions and version pinning. With <a href="https:&#x2F;&#x2F;registry.terraform.io&#x2F;modules&#x2F;giner&#x2F;repo&#x2F;github" rel="nofollow">https:&#x2F;&#x2F;registry.terraform.io&#x2F;modules&#x2F;giner&#x2F;repo&#x2F;github</a> you can manage all repos together with workflows.

Write a script that works anywhere and execute it via your ci tooling as a thin wrapper

I just don&#x27;t understand why people have gone down this ci in the cloud train. I&#x27;ve been using some form of build scripts locally and then just extended that to run on jenkins as a standard flow for the longest time and reproducibility is key. It always will be. If you just have a generic script or tool that builds your project and people can just pick it up and read it then it&#x27;s going to be way easier to adopt and debug. Stop trying to use platforms with a dsl, with breaking down pieces in their proprietary way. It makes no sense.

Sidenote --<p>What a cool looking website. What kind of tool do you need to create those animations?

I am building Toolhouse.ai and I&#x27;ve had headaches with Github actions but luckily I ask the AI to help out when I am trying to do something. My biggest annoyance is that it&#x27;s oddly hard to debug things without running it. Even the Github Actions syntax helper on VSCode isn&#x27;t super helpful. I have recently discovered `act`[1] and I will be investing time to using it because it really * hopefully * makes a difference.<p>[1]<a href="https:&#x2F;&#x2F;github.com&#x2F;nektos&#x2F;act" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;nektos&#x2F;act</a>

I try to use as little of GHA specific things as possible. Use it as a runner but don&#x27;t lock yourself into the platform. I want to be able to develop and run the CI outside GHA thank you very much.

It seems that whenever something becomes good enough to become popular, you start the clock on people starting to write about how it&#x27;s the worst thing ever.

On a related note - how are people tracking the absolute thicket of permissions quirks that is Github&#x27;s various secrets, tokens &amp; repo permissions?

Worst part of GitHub Actions? Its encouragement of the total misuse of containers as what have essentially become installer scripts - complete with all the flakiness you&#x27;d imagine.<p>Go look at your workflows and see how much of the runtime is spent running installers upon installers for various languages, package managers and so on. Containers were not supposed to be like this.

Most &quot;CI&quot; platforms suck in some way. I attribute it to a mix of misaligned incentives (less efficient pipelines, more premium-rate CPU cycles to resell, lockin, etc.) and the fact that it&#x27;s actually just a hard problem.<p>See: <a href="https:&#x2F;&#x2F;packetlost.dev&#x2F;Why%20Does%20CI%20Suck" rel="nofollow">https:&#x2F;&#x2F;packetlost.dev&#x2F;Why%20Does%20CI%20Suck</a>

I&#x27;ve found Actions and Codespaces to be different from one another. Actions once recently came with a borked gcc compiler, which failed to build some code that was fine before, and there was <i>no convenient way to debug this</i>. As in no way for me to spin up exactly the same GH Actions environment in Codespaces.<p>Why not align these tools? Then there might be less pain. What a good idea.

I was updating an old action last night to update gh pages and it’s from peaceiris. And it’s not bad, it did the job. But it feels kinda weird.

If you want an easy solution for GitHub Actions security, check out Garnet.ai (formerly listen.dev). They were built for GitHub first. And it’s free for single projects - <a href="https:&#x2F;&#x2F;dashboard.listen.dev&#x2F;" rel="nofollow">https:&#x2F;&#x2F;dashboard.listen.dev&#x2F;</a>.

At [previous company], we initially required branches to be up to date with main. Since it was a relatively big mono repo, it slowed down productivity quite a bit.<p>Eventually we tried dropping that requirement and instead relied on testing main before deploying to production. It sped us up again, and main never broke because of bad merges while I was there.

Modern CI tends to grow features until it&#x27;s reproduced the all the features of a build system. You then end up with the actual logic of your build smeared across mutiple layers. Or contorting your CI setup to allow the build system to do caching etc properly.

Not my experience, I have done fairly complex things like building &#x2F; releasing a whitelabel ios and android app which is branded in ci per customer.<p>Like othes have suggested, keep the actions simple by having lots of scripts which you can iterate on locally and making the actions dump to just run the scripts

I have a problem with the Github Actions documentation. There is a lot of it, but it feels as though it was written from a &quot;product&quot; perspective, to explain how to use the product.<p>None of it usefully explains how GHA works from the ground up, in a way that would help me solve problems I encounter.

Did you consider Pkl instead of Yaml to write your GitHub actions?<p><a href="https:&#x2F;&#x2F;github.com&#x2F;StefMa&#x2F;pkl-gha" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;StefMa&#x2F;pkl-gha</a><p>It could save you already some time.

One thing I found useful was writing a runner for giteas actions CI which is similar to GHA. When you dig down and ask &quot;what is ACTUALLY happening to run this job&quot; then a lot of things such as the docker entrypoint not being modifiable make perfect sense.

I still love my Jenkins Pipelines, Groovy shared libraries, and Bash scripts. It may be old, the UI may be a little crusty, but it&#x27;s well understood, it just works, and I don&#x27;t think much about the tool anymore.

This was actually an extremely valuable article for me. I was unaware of act, the tool to test GH workflows, and your personal flow of using a separate branch to troubleshot the yaml code makes perfect sense.

I wanted to try dagger to solve some of these issues (<a href="https:&#x2F;&#x2F;github.com&#x2F;dagger&#x2F;dagger" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;dagger&#x2F;dagger</a>) anyone has feedback on it?

Is there a decent setup that one can run on their own server(s) instead? Because I&#x27;d much rather have a dedicated server sitting in a closet connected to fiber whose only job it is to be a CI runner.

To get around the horror that is YAML, I wholeheartedly recommend writing GitHub workflows in CUE and generating the required YAML out of them. I&#x27;m hopefully never going back to writing YAML myself!

Github Actions are simple things for tasks. I do btw also not like them.<p>But, there are so many red flags in this post. Clearly this corp does not know how to build, test and release professional software.

I&#x27;ll take on the occasion to ask the HN crowd: have you noticed that caches, on top of being limited to 10GB, don&#x27;t seem to expire as advertized, in a LRU manner?

Github is too busy dumping all their engineering power into more useless copilot features than actually doing anything to improve their platform.

On the pain side: is it &quot;official&quot; that github issues cannot be posted anymore with noscript&#x2F;basic (x)html browsers?

I can relate to this pain. Isn&#x27;t gitlab CI better at this especially the documentation and simplicity of it?

I can’t believe Forgejo ever thought it was a good idea to try &amp; copy this nonsense. Rather than trying to be some FOSS MS GitHub clone, why not pitch that they can do things better—such as not having YAML spaghetti for CI.<p>I hope Actions stays bad tho. We need more folks to get off proprietary code forges for their open source projects—&amp; a better CI + a better review model (PRs are awful) are 2 very low-hanging fruit that would entice folks off of the platform not for the philosophical reasons such as not supporting US corporations or endangering contributor privacy by making them agree to Microsoft’s ToS, but for technical superiority on the platform itself.

I really urge people to try Cirrus CI, it&#x27;s almost unknown but it has pretty amazing features!

we are working on a platform <a href="https:&#x2F;&#x2F;yeet.cx" rel="nofollow">https:&#x2F;&#x2F;yeet.cx</a> that lets you audit github actions pretty easily.<p>you just activate some probes and write SQL queries to sift through the information.

I just wish the default wasn&#x27;t bash. GHA with pwsh is a much better experience.

Building in the cloud seems like the worst possible thing you could do.

Half these problems and complexity could be solved with precommit hooks.

Has anyone used argo on kubernetes to automate their CI pipeline?

As someone who&#x27;s been using it at very large scale, I still miss gitlab but I think they are not that bad.<p>But two major pains I did not see : the atrocious UI and the pricing.<p>Their pricing model goes against any good practice, as it counts a minute for any job even if it runs for 2 seconds. Let&#x27;s say you run 100jobs in parallel and they all take 30sec. You will pay 100 minutes instead of 50. Now translate this to an enterprise operating at a big scale and I assure you have seen crazy differences between actual time and billable time.

Keeping the tech stack simple helps a lot with the CI&#x2F;CD space. I still prefer to use custom tools that are part of the project source so I don&#x27;t get locked in. This is like EC2 vs FaaS for me. I&#x27;ll take the vanilla abstraction please.<p>Most of the time, I&#x27;m just running dotnet build to a zip file and s3 bucket. Then, some code or script picks it up on the other side. Things get much trickier when you&#x27;re using multiple services, languages, runtimes, database technologies, etc.

Git is a pain

Once again, Rails is ahead of the curve on this:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;rails&#x2F;rails&#x2F;pull&#x2F;54693" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;rails&#x2F;rails&#x2F;pull&#x2F;54693</a>

Off-topic but DALL-E has turned the web into slop-city. What a mess. Everything looks the same, cheap and ugly and stupid.

tldr but: don&#x27;t use GitHub Actions. Its a mess, the availability is often atrocious, and the UI around it is _still_ as clunky as when they first rolled it out many years ago.<p>There are better solutions out there.