GitHub was down

I&#x27;ve never seen an outage this big. Even the homepage doesn&#x27;t load. We&#x27;ve had recurrent issues with Actions not running, but this seems a lot bigger.<p>The status page says all is well, though: <a href="https:&#x2F;&#x2F;www.githubstatus.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.githubstatus.com&#x2F;</a>. Hilarious.

The timing is pretty uncanny. I just deployed a github page and had a DNS issue because I configured it wrong. I hit &quot;check again&quot; and github went down.<p>Hope I don&#x27;t appear in the incident report.

<a href="https:&#x2F;&#x2F;www.githubstatus.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.githubstatus.com&#x2F;</a> reports no problems, but it&#x27;s clearly down for a lot of people (including me).

@dang <a href="https:&#x2F;&#x2F;www.githubstatus.com&#x2F;incidents&#x2F;kz4khcgdsfdv" rel="nofollow">https:&#x2F;&#x2F;www.githubstatus.com&#x2F;incidents&#x2F;kz4khcgdsfdv</a> is probably a better link for this submission now

The mobile app on iOS is a 503 with<p>```<p>Received a 503 error. Data returned as a String was: &lt;!DOCTYPE html&gt; &lt;!- -<p>Hello future GitHubber! I bet you&#x27;re here to remove those nasty inline styles, DRY up these templates and make &#x27;em nice and re-usable, right?<p>Please, don&#x27;t. <a href="https:&#x2F;&#x2F;github.co" rel="nofollow">https:&#x2F;&#x2F;github.co</a>...<p>```<p>That&#x27;s where it&#x27;s cut off on my screen.<p>Curious what the link is :)<p>I like to think, someone did.

Could it have been brought down intentionally? Related to this?<p><a href="https:&#x2F;&#x2F;www.bleepingcomputer.com&#x2F;news&#x2F;security&#x2F;github-actions-artifacts-found-leaking-auth-tokens-in-popular-projects&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.bleepingcomputer.com&#x2F;news&#x2F;security&#x2F;github-action...</a>

All the AI-native developers are twiddling their thumbs because Copilot is out of office.

for everyone complaining about the status page - status pages are normally operated by hand by design, and will rarely reflect things in real-time.<p>give the poor github ops folks a second to get things moving.

It&#x27;s 00:16, just about to go to bed, I ran `git push` and it&#x27;s not working. Check Github, says it&#x27;s down, I think it&#x27;s only me, maybe I&#x27;m blocked, Github can&#x27;t be down. Come here to check and it&#x27;s down for everyone, such a relief.

The macho unicorn is kinda dope though.

RIP to all those who host their websites on GitHub pages :(

I see more and more people use less Github, but some other git solutions. I am afraid to think what to do when GitHub is down for hours (need to learn maillists?).<p>Another reason is that MS may be in phase when it will ask to pay for using GitHub just for reads (rate limiter).

Services that explicitly needed the API were also down, and it wasn&#x27;t pretty. For example: Minecraft Mod packs that rely on SerializationIsBad all went kerplunk! I&#x27;m sure a lot of people were scratching their heads yesterday wondering why they couldn&#x27;t do anything for a time.<p>What made me laugh though was when the &quot;X is functioning normally&quot; immediately followed by &quot;X is degraded, continuing to monitor&quot; messages that kept popping up then right back to &quot;normal&quot; again, all in the same 30 second timespan... made me giggle

<a href="https:&#x2F;&#x2F;www.githubstatus.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.githubstatus.com&#x2F;</a><p>This is a pretty good place to check. The lag is pretty minimal traditionally.<p>At the time of posting everything is broken.

Unfortunately, outages happen... This situation is a very good reminder of why having backups and a solid Disaster Recovery plan is crucial. Of course, it’s easy to assume that cloud services are always up, but we should never forget about outages. Setting up automated backups for repos and metadata can save a ton of headaches when things go wrong. Plus, having a Disaster Recovery plan means you’re not stuck waiting for the service to come back online—you can keep working with minimal disruption.<p>Here is a good article on how to prepare for the situations like that, when GitHub is down: <a href="https:&#x2F;&#x2F;gitprotect.io&#x2F;blog&#x2F;github-restore-and-github-disaster-recovery-scenarios-and-use-cases&#x2F;" rel="nofollow">https:&#x2F;&#x2F;gitprotect.io&#x2F;blog&#x2F;github-restore-and-github-disaste...</a>

Things seems to be ack&#x27;ed: ``` Investigating - We are investigating reports of degraded availability for Actions, Pages and Pull Requests Aug 14, 2024 - 23:11 UTC ```

Status page, like usually all green -&gt; <a href="https:&#x2F;&#x2F;www.githubstatus.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.githubstatus.com&#x2F;</a>

This was not the first or last GitHub outage. Unfortunately, this was a big one. That why now it is even more important to have backups of your work and restore capabilities in case of a scenario like this outage. This article sheds light on the importance of backups along with the best practices to follow:<p><a href="https:&#x2F;&#x2F;gitprotect.io&#x2F;blog&#x2F;github-backup-best-practices&#x2F;" rel="nofollow">https:&#x2F;&#x2F;gitprotect.io&#x2F;blog&#x2F;github-backup-best-practices&#x2F;</a>

Yes. <a href="https:&#x2F;&#x2F;www.githubstatus.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.githubstatus.com&#x2F;</a>

This will have a fun post mortem

Me: I think I&#x27;ll update nixos*<p>Nix: <i>barfs voluminous errors I&#x27;ve never seen before</i><p>Me: whaaaat the farrrrk<p>* nixos updates are pulled from a github repo

Its all down according to <a href="https:&#x2F;&#x2F;www.githubstatus.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.githubstatus.com&#x2F;</a><p><pre><code> Update - Issues is experiencing degraded availability. We are continuing to investigate. Aug 14, 2024 - 23:19 UTC Update - Git Operations is experiencing degraded availability. We are continuing to investigate. Aug 14, 2024 - 23:19 UTC Update - Packages is experiencing degraded availability. We are continuing to investigate. Aug 14, 2024 - 23:18 UTC Update - Copilot is experiencing degraded availability. We are continuing to investigate. Aug 14, 2024 - 23:13 UTC Update - Pages is experiencing degraded availability. We are continuing to investigate. Aug 14, 2024 - 23:12 UTC</code></pre>

A coworker and I just had to use `git format-patch` and `git am` to exchange work. Git is super cool!

I&#x27;m wondering why this isn&#x27;t on the front page? It has a lot of points in 23 minutes.

Wonder if this is related to the big cyberattack on Iran earlier today <a href="https:&#x2F;&#x2F;www.jpost.com&#x2F;breaking-news&#x2F;article-814715" rel="nofollow">https:&#x2F;&#x2F;www.jpost.com&#x2F;breaking-news&#x2F;article-814715</a>

Down in the Dominican Republic as well, was just trying to commit and end my day

Yes, everything is down <a href="https:&#x2F;&#x2F;www.githubstatus.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.githubstatus.com&#x2F;</a>

Feels bad to have one&#x27;s job interrupted. Looking on the bright side this is the excuse I needed to check out Radicle...

Yes it went down about 5 mins ago, I got the angry unicorn. Since then the status page is increasingly red.<p>Seeing it all kind of went sideways at the same time, my money is on the typical load balancer config rollout snafu.<p><pre><code> &quot;As part of a routine configuration deploym...&quot; [splat]</code></pre>

And so goes all your packages, private repositories, pages, AI intern copilot bot and Github Actions; and soon your AI models once you host them there - all being unavailable and going down with GitHub.<p>Time to consider self-hosting like the old days instead of this weekly chaos at GitHub.

Who&#x27;s the Bozo Doofus maintainer? <a href="https:&#x2F;&#x2F;yhbt.net&#x2F;unicorn&#x2F;LATEST" rel="nofollow">https:&#x2F;&#x2F;yhbt.net&#x2F;unicorn&#x2F;LATEST</a>. I love that we can still see Unicorn in action. I rarely had problems with it back in the day.

Cause seems to be database related per most recent update (23:29 UTC):<p>&gt; We are experiencing interruptions in multiple public GitHub services. We suspect the impact is due to a database infrastructure related change that we are working on rolling back. Aug 14, 2024 - 23:29 UTC

@dang maybe the link could now be updated with this one <a href="https:&#x2F;&#x2F;www.githubstatus.com&#x2F;incidents&#x2F;kz4khcgdsfdv" rel="nofollow">https:&#x2F;&#x2F;www.githubstatus.com&#x2F;incidents&#x2F;kz4khcgdsfdv</a>

time to go to bed then. Wasn&#x27;t getting any useful work done any more anyways...

Latest update at 23:29 UTC says: &quot;We are experiencing interruptions in multiple public GitHub services. We suspect the impact is due to a database infrastructure related change that we are working on rolling back.&quot;

Aug 14, 2024 - 23:29 UTC Update - We are experiencing interruptions in multiple public GitHub services. We suspect the impact is due to a database infrastructure related change that we are working on rolling back.

We are experiencing interruptions in multiple public GitHub services. We suspect the impact is due to a database infrastructure related change that we are working on rolling back. hope it is back up soon

We are experiencing interruptions in multiple public GitHub services. We suspect the impact is due to a database infrastructure related change that we are working on rolling back. hope it comes back soon

00:30 UTC Resolved <a href="https:&#x2F;&#x2F;www.githubstatus.com&#x2F;incidents&#x2F;kz4khcgdsfdv" rel="nofollow">https:&#x2F;&#x2F;www.githubstatus.com&#x2F;incidents&#x2F;kz4khcgdsfdv</a>

There goes Pages, there goes the CDN for release artifacts, there goes any package manager hosting repositories on GitHub. Is this outage just contained to github or is it an Azure outage?

Down in Australia as well

Yep, angry unicorn. If the copilot debacle wasn&#x27;t reason enough to make people migrate or diversify the code repo efforts with, let&#x27;s say, GitLab, this should.

It feels so wrong that there are so many blogs and websites that are based on GH Pages and they all died at once…<p>Seems like they’re back up though. Or at least the Rust blog is back up.

This reminds me that for some reason I am logged into my gaming machine&#x27;s windows store with my GitHub account thanks to the bizarre way that microsoft do auth.

Down Detector link:<p><a href="https:&#x2F;&#x2F;downdetector.com&#x2F;status&#x2F;github&#x2F;" rel="nofollow">https:&#x2F;&#x2F;downdetector.com&#x2F;status&#x2F;github&#x2F;</a>

Even GitHub-hosted Pages are down — <a href="https:&#x2F;&#x2F;prql-lang.org&#x2F;" rel="nofollow">https:&#x2F;&#x2F;prql-lang.org&#x2F;</a> is also a unicorn

I sure wish this had happened before I logged off from work for the day...<p>&quot;Why isn&#x27;t this project done yet?&quot;<p>&quot;Didn&#x27;t you hear? GitHub is down!&quot;<p>and I get to go out for a long lunch

Nothing useful on the status page: <a href="https:&#x2F;&#x2F;www.githubstatus.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.githubstatus.com&#x2F;</a>

I am happy that my project is pushed also to codeberg.

This is weird. I&#x27;ve been using Github all night (in France) and didn&#x27;t notice anything was wrong. Was the outage in North America?

Yes : <a href="https:&#x2F;&#x2F;github.com&#x2F;psf&#x2F;black&#x2F;">https:&#x2F;&#x2F;github.com&#x2F;psf&#x2F;black&#x2F;</a> is 502

Interesting that CoPilot is down as well. I would have assumed it was really only part of GitHub as a branding&#x2F;marketing thing.

Investigating - We are investigating reports of degraded availability for Actions, Pages and Pull Requests Aug 14, 2024 - 23:11 UTC

Yes <a href="https:&#x2F;&#x2F;www.githubstatus.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.githubstatus.com&#x2F;</a>

Would this explain why &quot;npm install next-sanity&quot; doesn&#x27;t work properly, or am I hitting a user error?

Welp, that’s as good a time as any to call it a day!<p>Good luck to the devs and dev-analogues involved in getting the ship righted.

Can&#x27;t wait for the writeup! So many services down at once... Something very interesting must have happened

They were manually ran a hot patch on the distributed production database and forgot to use a transaction

I&#x27;ve never seen such a serious outage before. Even GitHub Pages hosted sites aren&#x27;t accessible.

Thank goodness for HN status reports.

And other services like copilot...<p>[error] [auth] Response content-type is text&#x2F;html; charset=utf-8 (status=503)

Seems like sites based on GH Pages were down, but are back up (i.e. the Rust blog).

This is my first time seeing the angry unicorn! Hopefully it’ll be gone soon :(

Time to go out and see people

I swear even my VSCode intellisense is broken now... Rip to a real one.

I think it maybe global but at least it is down in US-East (for sure).

Down in Vancouver, Canada

Seems to be back online.

So that&#x27;s why I haven&#x27;t heard back on my applications

The status page should have a button &quot;Report Outage&quot;.

Down for me as well. Thought my SSH agent was broken.

Feels like Github is down more often than Gitlab

Seem to be up again, I also wonder what is was.

Maybe it&#x27;s fixed already? Works for me.

I love how the same people who try to drag me towards using Git are the only people who seem to have serious problems working on their code when a website goes down.

Status page showing a complete outage now

What&#x27;ll it be this time? DNS or BGP?

given that it seems like the entire thing is busted, can anyone explain how the unicorn page is being served?

and I was in the middle of commiting a hot fix D:, I had to push the image directly to the registry D:

Yes, github is not working right now

on X @githubstatus seems to be getting regular updates &#x2F; automated messages around impact

Wtf, thought it was me alone!

it crashed the second i opened a github rep for an old plugin for blockbench

They manually ran a patch query on the distributed production database and forgot to use a transaction

Sure is.

it crashed the second i opened this old plugin for blockbench

curious if their layoffs last year had the intended impact

just wanted to do one last final commit :D good timing

totally down -.- Cannot access from Hong Kong OMG...

Sure looks like it!

Back online for me

and... we&#x27;re back, at least in Japan region

Wonder if they’ve had worse uptime after moving to Azure

Github mirrors?

down! unicorn!<p>fatal: unable to access: 502<p>cli, web, and iOS app :-&#x2F;

Yep

Imagine everyone having their site hosted on gh pages. Now imagine artifacts for system update on github. Hello NixOS filesystem!

&quot;We suspect the impact is due to a database infrastructure related change that we are working on rolling back.&quot;

Imagine everyone having their site on gh pages, and artifacts for OS startup on here. Hello, NixOS...

down in phx az

back to live

Wonder how much of this is to blame on copilot generated code lol

back up

Yet again, this shows how useless GitHub status page is.

totally down -.-

...and we&#x27;re back

down for me

Love that HN is a better status page for dev services than most companies can manage to provide. Knew I&#x27;d find it here but on the front page within 3 minutes is impressive.

A reminder of how centralized and dependent the whole industry has become on GH, which is ironic, considering that git itself is designed to be decentralized.<p>Good opportunity to think about mirroring your repos somewhere else like Gitea or Gitlab.

I wonder how much it would&#x27;ve taken to keep github running without Microsoft buying them and running them into the ground like this.

In the last 5 minutes too, wow.

They should have used Kardinal: <a href="https:&#x2F;&#x2F;github.com&#x2F;kurtosis-tech&#x2F;kardinal">https:&#x2F;&#x2F;github.com&#x2F;kurtosis-tech&#x2F;kardinal</a>

GH Ops team be like<p>Senior: Ah found it! Let&#x27;s just rollback one revision on the db. Newguy: let me fix this! `kubectl rollout undo ... --to-revision=1` Newguy: Ok, Started rollback to revision one! Senior: Uh-oh..