Update on 1/28 service outage

Yesterday I was being a bit of an ass to a few people about how "the whole point of using git is so that we can do decentralized code management and why these dependencies were being pulled from our private github if the could be sent point to point yadda yadda yadda". Then they proceeded to go over the list of package managers and dependencies we used and I had to shut up. Even when we host our own Docker Hub and package managers (we do), if you dig far enough, you can find some dependency of a dependency of dependency that relies on GitHub. Brew/npm/build script/whatever. It is crazy how everything has changed so much in the past few years. GitHub went from something that was really nice to have to a core requirement for complex systems that rely heavily on open source.

Am I the only one who is a little shocked that a power outage could have such a huge effect and bring them down for so long? I'm not an infrastructure guy, and I don't know anything about Github's systems, but aren't data center power outages pretty much exactly the kind of thing you plan for with multi-region failover and whatnot. Is it actually frighteningly easy for kind of to happen despite following best practices? Or is it more likely that there's more to the story than what they're sharing now?

Here&#x27;s the only page I could quickly find on Github&#x27;s architecture for those interested:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;blog&#x2F;530-how-we-made-github-fast" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;blog&#x2F;530-how-we-made-github-fast</a><p>This looks like a single datacenter. I don&#x27;t see anything here indicating high availability or other datacenters. You&#x27;ll usually spot either an outright mention of it or certain components&#x2F;setups common in it. They might have updated their stuff for redundancy since then. However, if it&#x27;s same architecture, then the reason for the downtime might be intentional design where only a single datacenter has to go down.<p>Might be fine given how people apparently use the service. It&#x27;s just good to know that this is the case so users can factor that into how they use the product and have a way of working around the expected downtime if it&#x27;s critical to do so.

&quot;Millions of people and businesses depend on GitHub&quot;<p>Well, we shouldn&#x27;t depend on it so much.<p>I shudder at the thought what an outage of GitHub would mean for our company. This time, we were lucky as it was during the night in Europe.<p>Unfortunately, I don&#x27;t have the power to test this scenario in our company.

It&#x27;s one thing when one temporarily loses access to remote repositories for pushes. Quite bearable, because you can exchange code across your corporate network using patches and whatnot. And it&#x27;s totally different when you cannot friggin <i>build</i> anything because package managers grab dependencies directly off of GitHub.

Not much detail here. A more thorough postmortem would give me more confidence they can recover from another similar issue. Hoping to see one soon.

You can see the cascade effect on their status page graphs: <a href="https:&#x2F;&#x2F;status.github.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;status.github.com&#x2F;</a>

This post makes it sound like Github has it&#x27;s own data centers and power infrastructure structure, this is definitely news to me.. I&#x27;d presumed co-lo at best.

Github doesn&#x27;t deploy their services in multiple az&#x27;s?

it seriously makes me lol that people are upset, or surprised, that an internet service went down for a couple of hours. a couple of hours! get some perspective please. go for a walk, get a tasty burrito, try a new brand of hot sauce.<p>&quot;why didn&#x27;t they do X, Y, or Z&quot;<p>the answer in every case is it&#x27;s extremely expensive, or extremely hard to do, or both. you want a reason, there&#x27;s the reason. maybe they&#x27;ll fix it. maybe they won&#x27;t. next question.<p>make your own backups and redundant systems. &quot;but github is so critical!&quot; -- even more reason to have a backup. bad shit happens in this world. even to good people. prepare or suffer the consequences.

Maybe I&#x27;m ignorant, but why do companies rely on github? Why not just host it in-house? If there&#x27;s power outage in the office then everything would be down anyways, right?

Would it be possible for a cross between Git and Torrents? Rather than having a central server to pull&#x2F;push from, instead the server would provide a list of clients. If the server goes down, the list is still available, and so people who depend on it would be able to communicate.

Why is it so hard for us to distribute our dependencies? Hash the package to a sha and put t anywhere on the Internet. Then we just need a service that holds and updates the locations of the hashes and we can fetch them anywhere.

For those that have been affected by this, what parts of your process were disrupted? I&#x27;ve read, so far:<p><pre><code> * Build fails due to unreachable dependencies hosted by GitHub * Development process depends on PRs</code></pre>

Chinese DDoS? Somehow I don&#x27;t buy power going out at a server farm.

It&#x27;s always scary when a cloud service you rely on goes down but great to see GitHub recover. Well done!

Various date&#x2F;time formats across the world bringing me to the knees. If 1&#x2F;28 outage was _that_ rough 2&#x2F;28 would be twice as bad and 28&#x2F;28 would feel like armageddon maybe?

I recently read a blog post from Github about them operating their own datacenter <a href="http:&#x2F;&#x2F;githubengineering.com&#x2F;githubs-metal-cloud&#x2F;" rel="nofollow">http:&#x2F;&#x2F;githubengineering.com&#x2F;githubs-metal-cloud&#x2F;</a><p>Im not positive, but it sounds like a fairly recent switch from a cloud provider to their own datacenter. If thats the case, Id expect a number of outages to come in the following months.