Incident with Issues and Pull Requests

GitHub outages are very reliable, as I live in Europe and they always come in the afternoon they&#x27;re a great reminder to go get lunch<p>It&#x27;s a feature, not a bug!

What is going on over there? Third day in a row is... kind of impressive.

I&#x27;m considering host a gitea instance backup all of my repos.<p>I have an important fix that need to be deployed right now but there is no way to deploy it in a normal way with our CI which one was setup with Github Action. Fortunately I have a instruction to bypass CI and build the source by myself.<p>But again, Github defeat me because our release workflows are depend on GitOps which are effected by Github issue. Ahhhhhhhhhh I have to build the docker image, push it to ECR then update a YAML template to make EKS apply the new changes<p>It&#x27;s 9PM in my timezone and I&#x27;m waiting for my patches are up. A frustrating incident

Daily reminder that <a href="https:&#x2F;&#x2F;github.com&#x2F;MichaelMure&#x2F;git-bug">https:&#x2F;&#x2F;github.com&#x2F;MichaelMure&#x2F;git-bug</a> could use some help :-D

So nice to meet you all here again, gents - this is becoming a regular thing.

We are back to Windows 95, which reliably crashed after 49.7 days:<p><a href="https:&#x2F;&#x2F;www.cnet.com&#x2F;culture&#x2F;windows-may-crash-after-49-7-days&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.cnet.com&#x2F;culture&#x2F;windows-may-crash-after-49-7-da...</a>

&gt; Codespaces is experiencing degraded performance. We are continuing to investigate.<p>Imagine not only not being able to push your code, but also not even being able to _write your code_ at all. And so many orgs rely on Actions to even be able to deploy. Geez. I personally believe that the cloud sucks.

Not just issues and PRs - I had to try multiple times before I was able to successfully push code to a repo over SSH.<p>This is the error I was seeing:<p><pre><code> ERROR: fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists.</code></pre>

Anecdotal - I&#x27;ve been using Gitlab for a few years on some projects, and haven&#x27;t experienced any downtime issues with them of this magnitude.

The temptation to create `isgithubup.com` and return &quot;surprisingly yes.&quot; on the rare occasion it actually is up.

Three days in a row of outages, in less than a week of unreliability after yesterday&#x27;s downtime of GitHub Actions [0].<p>Really at this point, you just might as well consider self hosting and it is looking very chronic with GitHub falling apart and self-hosting was indeed the sensible idea just like how the other open source projects have done for years.<p>GitHub is going just great, and centralizing everything to GitHub really was a good idea wasn&#x27;t it? [1] &#x2F;s<p>[0] <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=35887029" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=35887029</a><p>[1] <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=22867803" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=22867803</a>

Shout out to all devs that put the deps of their programs on GH...<p>Right now I can ignore that my PRs show 500 error, or old code best case scenario.<p>But... I cannot build and ship due to project&#x27;s dependency depending on some stuff hosted on GH.

Hugs to the GitHub Opps and SRE teams right now!<p>Also hugs to any Devs, Opps or SREs directly effected by this outside GitHub.<p>Looking forward to a post-mortem on the last few days, I&#x27;m sure it will be a really interesting read.

Third day in a row this has affected my productivity

500&#x27;s on every repo page + CLI errors on pull &#x2F; push. Completely useless at the moment.

Anyone experienced with asking for service credits for enterprise customers? It looks like you have to prove that they have missed their SLAs

I&#x27;ve had some actions queued for multiple days now on certain repos, but not others. I&#x27;ve cancelled them and restarted them during the green status intervals but they all go back to &quot;Queued&quot;. I&#x27;ve also cancelled them and then made slight documentation tweaks to get new commit hashes on the branches and it still goes to queued.

Just lost a merge commit to dev&#x2F;null. This is getting tiresome

I&#x27;m going to self-host my git repos. Any recommendations?<p>The git+nginx would suffice but it does not offer GUI. I need one to see the changes proposed (aka PRs).<p>Gitea is nice, but a bit overkill for my needs. I don&#x27;t need CI, files hosting, issues, team members, releases, wiki, forking&#x2F;watching&#x2F;staring, etc.

this is the third day in a row this is a problem yet they&#x27;re framing it as a new outage.

Loading github.com is returning a 500 for me currently, so seems like more than just issues&#x2F;pull requests. Also seeing actions fail with 500s on assorted steps.

It started for me a few minutes before the status page showed something. Which is understandable, of course.<p>But strange that it keeps happening almost every day now.

Does anyone host their own git repos in an enterprise environment? How do you do it and what are some good resources for learning how to do this?

This gives me some the fail-whale vibes. But the blue bird didn&#x27;t fail like this these days, strangely.

Have they said what has been up lately?

I feel that with dynamic like this, someone could make a page showing the number of days since the last github incident.<p>It would show a very prominent zero and be a static page with no logic whatsoever.

Yup, hitting me and all over twitter: <a href="https:&#x2F;&#x2F;twitter.com&#x2F;search?q=github" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;search?q=github</a>

Trying to access repos is returning 500 for me also

Maybe they should put CoPilot in charge of ops...

I wasn&#x27;t even able to get to repos or users&#x2F;orgs for a while (though as I write this it seems like that is coming back).

and i thought i broke `master`...

git says my code pushed fine and everything is synced, but Github is not showing any of my changes.<p>Edit: 10 minutes later, the Github finally shows the push, but triggers still aren&#x27;t working.<p>Edit #2: Things are working normally now.

Another day, another outage. I&#x27;m getting 500 for any repo.

Thee days in a row!! Is this the result of all the job cuts?

Is it time to move back to Jenkins? :X

I cannot even push code

Here we go again…

Back to biz

On the bright side, maybe, just maybe the open source community realises that such centralisation might not be the best solution for hosting code.