I've never seen an outage this big. Even the homepage doesn't load. We've had recurrent issues with Actions not running, but this seems a lot bigger.<p>The status page says all is well, though: <a href="https://www.githubstatus.com/" rel="nofollow">https://www.githubstatus.com/</a>. Hilarious.
The timing is pretty uncanny. I just deployed a github page and had a DNS issue because I configured it wrong. I hit "check again" and github went down.<p>Hope I don't appear in the incident report.
<a href="https://www.githubstatus.com/" rel="nofollow">https://www.githubstatus.com/</a> reports no problems, but it's clearly down for a lot of people (including me).
@dang <a href="https://www.githubstatus.com/incidents/kz4khcgdsfdv" rel="nofollow">https://www.githubstatus.com/incidents/kz4khcgdsfdv</a> is probably a better link for this submission now
The mobile app on iOS is a 503 with<p>```<p>Received a 503 error. Data returned as a String was: <!DOCTYPE html>
<!- -<p>Hello future GitHubber! I bet you're here to remove those nasty inline styles, DRY up these templates and make 'em nice and re-usable, right?<p>Please, don't. <a href="https://github.co" rel="nofollow">https://github.co</a>...<p>```<p>That's where it's cut off on my screen.<p>Curious what the link is :)<p>I like to think, someone did.
Could it have been brought down intentionally? Related to this?<p><a href="https://www.bleepingcomputer.com/news/security/github-actions-artifacts-found-leaking-auth-tokens-in-popular-projects/" rel="nofollow">https://www.bleepingcomputer.com/news/security/github-action...</a>
for everyone complaining about the status page - status pages are normally operated by hand by design, and will rarely reflect things in real-time.<p>give the poor github ops folks a second to get things moving.
It's 00:16, just about to go to bed, I ran `git push` and it's not working. Check Github, says it's down, I think it's only me, maybe I'm blocked, Github can't be down. Come here to check and it's down for everyone, such a relief.
I see more and more people use less Github, but some other git solutions.
I am afraid to think what to do when GitHub is down for hours (need to learn maillists?).<p>Another reason is that MS may be in phase when it will ask to pay for using GitHub just for reads (rate limiter).
Services that explicitly needed the API were also down, and it wasn't pretty. For example: Minecraft Mod packs that rely on SerializationIsBad all went kerplunk! I'm sure a lot of people were scratching their heads yesterday wondering why they couldn't do anything for a time.<p>What made me laugh though was when the "X is functioning normally" immediately followed by "X is degraded, continuing to monitor" messages that kept popping up then right back to "normal" again, all in the same 30 second timespan... made me giggle
<a href="https://www.githubstatus.com/" rel="nofollow">https://www.githubstatus.com/</a><p>This is a pretty good place to check. The lag is pretty minimal traditionally.<p>At the time of posting everything is broken.
Unfortunately, outages happen... This situation is a very good reminder of why having backups and a solid Disaster Recovery plan is crucial. Of course, it’s easy to assume that cloud services are always up, but we should never forget about outages. Setting up automated backups for repos and metadata can save a ton of headaches when things go wrong. Plus, having a Disaster Recovery plan means you’re not stuck waiting for the service to come back online—you can keep working with minimal disruption.<p>Here is a good article on how to prepare for the situations like that, when GitHub is down: <a href="https://gitprotect.io/blog/github-restore-and-github-disaster-recovery-scenarios-and-use-cases/" rel="nofollow">https://gitprotect.io/blog/github-restore-and-github-disaste...</a>
Things seems to be ack'ed:
```
Investigating - We are investigating reports of degraded availability for Actions, Pages and Pull Requests
Aug 14, 2024 - 23:11 UTC
```
Status page, like usually all green -> <a href="https://www.githubstatus.com/" rel="nofollow">https://www.githubstatus.com/</a>
This was not the first or last GitHub outage. Unfortunately, this was a big one. That why now it is even more important to have backups of your work and restore capabilities in case of a scenario like this outage. This article sheds light on the importance of backups along with the best practices to follow:<p><a href="https://gitprotect.io/blog/github-backup-best-practices/" rel="nofollow">https://gitprotect.io/blog/github-backup-best-practices/</a>
Me: I think I'll update nixos*<p>Nix: <i>barfs voluminous errors I've never seen before</i><p>Me: whaaaat the farrrrk<p>* nixos updates are pulled from a github repo
Its all down according to <a href="https://www.githubstatus.com/" rel="nofollow">https://www.githubstatus.com/</a><p><pre><code> Update - Issues is experiencing degraded availability. We are continuing to investigate.
Aug 14, 2024 - 23:19 UTC
Update - Git Operations is experiencing degraded availability. We are continuing to investigate.
Aug 14, 2024 - 23:19 UTC
Update - Packages is experiencing degraded availability. We are continuing to investigate.
Aug 14, 2024 - 23:18 UTC
Update - Copilot is experiencing degraded availability. We are continuing to investigate.
Aug 14, 2024 - 23:13 UTC
Update - Pages is experiencing degraded availability. We are continuing to investigate.
Aug 14, 2024 - 23:12 UTC</code></pre>
Wonder if this is related to the big cyberattack on Iran earlier today <a href="https://www.jpost.com/breaking-news/article-814715" rel="nofollow">https://www.jpost.com/breaking-news/article-814715</a>
Yes it went down about 5 mins ago, I got the angry unicorn. Since then the status page is increasingly red.<p>Seeing it all kind of went sideways at the same time, my money is on the typical load balancer config rollout snafu.<p><pre><code> "As part of a routine configuration deploym..." [splat]</code></pre>
And so goes all your packages, private repositories, pages, AI intern copilot bot and Github Actions; and soon your AI models once you host them there - all being unavailable and going down with GitHub.<p>Time to consider self-hosting like the old days instead of this weekly chaos at GitHub.
Who's the Bozo Doofus maintainer? <a href="https://yhbt.net/unicorn/LATEST" rel="nofollow">https://yhbt.net/unicorn/LATEST</a>. I love that we can still see Unicorn in action. I rarely had problems with it back in the day.
Cause seems to be database related per most recent update (23:29 UTC):<p>> We are experiencing interruptions in multiple public GitHub services. We suspect the impact is due to a database infrastructure related change that we are working on rolling back.
Aug 14, 2024 - 23:29 UTC
@dang maybe the link could now be updated with this one <a href="https://www.githubstatus.com/incidents/kz4khcgdsfdv" rel="nofollow">https://www.githubstatus.com/incidents/kz4khcgdsfdv</a>
Latest update at 23:29 UTC says: "We are experiencing interruptions in multiple public GitHub services. We suspect the impact is due to a database infrastructure related change that we are working on rolling back."
Aug 14, 2024 - 23:29 UTC
Update - We are experiencing interruptions in multiple public GitHub services. We suspect the impact is due to a database infrastructure related change that we are working on rolling back.
We are experiencing interruptions in multiple public GitHub services. We suspect the impact is due to a database infrastructure related change that we are working on rolling back.
hope it is back up soon
We are experiencing interruptions in multiple public GitHub services. We suspect the impact is due to a database infrastructure related change that we are working on rolling back.
hope it comes back soon
There goes Pages, there goes the CDN for release artifacts, there goes any package manager hosting repositories on GitHub. Is this outage just contained to github or is it an Azure outage?
Yep, angry unicorn.
If the copilot debacle wasn't reason enough to make people migrate or diversify the code repo efforts with, let's say, GitLab, this should.
It feels so wrong that there are so many blogs and websites that are based on GH Pages and they all died at once…<p>Seems like they’re back up though. Or at least the Rust blog is back up.
This reminds me that for some reason I am logged into my gaming machine's windows store with my GitHub account thanks to the bizarre way that microsoft do auth.
Down Detector link:<p><a href="https://downdetector.com/status/github/" rel="nofollow">https://downdetector.com/status/github/</a>
Even GitHub-hosted Pages are down — <a href="https://prql-lang.org/" rel="nofollow">https://prql-lang.org/</a> is also a unicorn
I sure wish this had happened before I logged off from work for the day...<p>"Why isn't this project done yet?"<p>"Didn't you hear? GitHub is down!"<p>and I get to go out for a long lunch
Nothing useful on the status page: <a href="https://www.githubstatus.com/" rel="nofollow">https://www.githubstatus.com/</a>
I love how the same people who try to drag me towards using Git are the only people who seem to have serious problems working on their code when a website goes down.
Love that HN is a better status page for dev services than most companies can manage to provide. Knew I'd find it here but on the front page within 3 minutes is impressive.
A reminder of how centralized and dependent the whole industry has become on GH, which is ironic, considering that git itself is designed to be decentralized.<p>Good opportunity to think about mirroring your repos somewhere else like Gitea or Gitlab.
They should have used Kardinal: <a href="https://github.com/kurtosis-tech/kardinal">https://github.com/kurtosis-tech/kardinal</a>
GH Ops team be like<p>Senior: Ah found it! Let's just rollback one revision on the db.
Newguy: let me fix this! `kubectl rollout undo ... --to-revision=1`
Newguy: Ok, Started rollback to revision one!
Senior: Uh-oh..