TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

GitHub: October 21 Incident Report

275 pointsby pietroalbiniover 6 years ago

21 comments

_verandaguyover 6 years ago
I saw some comments on reddit which highlighted a pretty serious problem - many orgs rely on github as a fully integrated CD platform, with everything from code hosting, to running CI hooks, to pushing to staging or prod.<p>It seems very unwise to have essentially your whole deployment process manager in the hands of an entity which you don&#x27;t only have no control over, but which has regularly been targeted in attacks by nation-state-level actors because of their role as a code hosting platform.<p>EDIT: GH hasn&#x27;t been targeted regularly, but it has been so historically, so this is a plausible thing which might happen again.
评论 #18274937 未加载
评论 #18276744 未加载
评论 #18280511 未加载
评论 #18284577 未加载
shashwat986over 6 years ago
Yeah, we&#x27;re still facing issues with, erm, Github issues.<p>Also, while they haven&#x27;t updated this blog post for a while, their status page has been very up-to-date and informative: <a href="https:&#x2F;&#x2F;status.github.com&#x2F;messages" rel="nofollow">https:&#x2F;&#x2F;status.github.com&#x2F;messages</a>
评论 #18273901 未加载
评论 #18273900 未加载
platiniumover 6 years ago
On the plus side, this disastrous calamity by Github really made me try out Gitlab and in the process, I will now set-up a second remote on my repo&#x27;s:<p><a href="https:&#x2F;&#x2F;stackoverflow.com&#x2F;questions&#x2F;11690709&#x2F;can-a-project-have-multiple-origins" rel="nofollow">https:&#x2F;&#x2F;stackoverflow.com&#x2F;questions&#x2F;11690709&#x2F;can-a-project-h...</a><p><pre><code> Quoted: &quot;Try adding a remote called &quot;github&quot; instead: $ git remote add github https:&#x2F;&#x2F;github.com&#x2F;Company_Name&#x2F;repository_name.git # push master to github $ git push github master # Push my-branch to github and set it to track github&#x2F;my-branch $ git push -u github my-branch # Make some existing branch track github instead of origin $ git branch --set-upstream other-branch github&#x2F;other-branch&quot; </code></pre> Actually, I don&#x27;t know why I pay for Github private repo&#x27;s.. I might as well set-up two origins, one at Gitlab and one at Bitbucket, for all my privates. Then keep Github as a public front-facing portal.
评论 #18275985 未加载
评论 #18276874 未加载
评论 #18277352 未加载
评论 #18276206 未加载
评论 #18276876 未加载
jbb67over 6 years ago
Ugh I detest the term &quot; abundance of caution &quot; as used in that message. Weasel words designed to stop you thinking about the problem as who wouldn&#x27;t want them to be overly cautious
评论 #18273714 未加载
vqngover 6 years ago
&gt; During this time, information displayed on GitHub.com is likely to appear out of date;<p>if the new data is not presented, users will typically retry which may result in duplicated new content.
评论 #18273640 未加载
diegoperiniover 6 years ago
As long as the same problem doesn&#x27;t occur again, I have no hard feelings against them.
Ajedi32over 6 years ago
Wow, is this still going on? I noticed this yesterday and figured it would be fixed in a few hours, but [GitHub&#x27;s status page][1] is still showing red. Is this the longest outage GitHub has had?<p>[1]: <a href="https:&#x2F;&#x2F;status.github.com&#x2F;messages" rel="nofollow">https:&#x2F;&#x2F;status.github.com&#x2F;messages</a>
LaserToyover 6 years ago
This incident report tells very little. I hope they release what actually happened and how it affected their services. And how they are going to avoid it in the future. Almost any issue can be publicly described as: stuff broke because of network.
评论 #18276245 未加载
评论 #18276001 未加载
nik736over 6 years ago
I don&#x27;t understand why code hosting platforms like GitHub, GitLab or BitBucket have so many issues regularly. Is there anything special about it?
评论 #18273681 未加载
评论 #18273661 未加载
评论 #18273656 未加载
评论 #18289696 未加载
评论 #18273639 未加载
评论 #18273824 未加载
评论 #18274402 未加载
评论 #18273641 未加载
dvfjsdhgfvover 6 years ago
It would be interested to know what was the cause of the network partition failure.
mr_toadover 6 years ago
Something that struck me yesterday as this started, is that Github isn’t really just a dvcs hosting solution, Github is a social network.<p>It’s easy to change the remote origin of a git repository. It’s not hard to migrate a project to Gitlab etc, and duplicate all the technical features of Github.<p>What’s really hard is replacing the social graph. If you’re a large project with a lot of contributors, onboarding everyone is not going to be easy. About as easy as convincing all your friends to stop using Facebook.
josteinkover 6 years ago
&gt; Further, this incident only impacted website metadata stored in our MySQL databases, such as issues and pull requests.<p>Basically the only things I was going to do on Github today :)
sneakover 6 years ago
Why bother telling people five incorrect estimates for service recovery?<p><a href="https:&#x2F;&#x2F;twitter.com&#x2F;sneakdotberlin&#x2F;status&#x2F;1054345379716558849" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;sneakdotberlin&#x2F;status&#x2F;105434537971655884...</a><p>After one or two wrong guesses it just shows that you’re making it up.
richardwhiukover 6 years ago
I hope they are going to provide a better RCA than that!
评论 #18273805 未加载
foxhopover 6 years ago
Don&#x27;t try creating a new repo, it will create part of the meta data but not allow you to see or use the repo and the repo name gets taken.
评论 #18274674 未加载
评论 #18274952 未加载
sneakover 6 years ago
GitHub also silently published a private repo of mine this week. I checked audit logs for both the owner user and the org and it didn’t show a permissions change anywhere.<p>Netlify has caused me to stop using GitHub Pages and between the clownshoes outage reports and the security issue I am now a GitLab user.<p>This is GitHub’s jump the shark episode. :(
评论 #18278089 未加载
wangyjxover 6 years ago
This was a head-on blow for MS?
k_over 6 years ago
Also discussed there: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=18271180" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=18271180</a>
mncharityover 6 years ago
As I rearrange today&#x27;s todo list, I recall wishing that I&#x27;d used the Microsoft purchase event to encourage folks to increase their familiarity with gitlab. So I now note:<p><a href="https:&#x2F;&#x2F;gitlab.com&#x2F;explore&#x2F;projects" rel="nofollow">https:&#x2F;&#x2F;gitlab.com&#x2F;explore&#x2F;projects</a> is a live feed of project activity, suitable for code surfing. It also sorts by stars and trending.
edejongover 6 years ago
Lets do a quick back of the envelope calculation:<p>Github reports 28,337,706 users by 2018-06-05 [1]. Lets assume 50% of these are active. Lets also assume that, due to the unavailability of GH, around 2 usable hours per developer are lost. Another assumption is that each developer contributes around 50 US$ per hour.<p>This means, this outage has cost us users: (28337706 * .5 * 2 * 50) = 1.351 billion US$.<p>Perhaps not use MySQL for such critical systems?<p>[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;search?q=type:user&amp;type=Users" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;search?q=type:user&amp;type=Users</a>
评论 #18274610 未加载
评论 #18274427 未加载
评论 #18274425 未加载
smilesmile2018over 6 years ago
GitHub team seems to be <i>VERY</i> unprofessional. 15 hour outage means ~99.82% availability which is extremely bad. 9 hours ago they also told that they would fix the problem within 2 hours... still not fixed!!!
评论 #18275870 未加载
评论 #18276023 未加载
评论 #18275883 未加载