HN was down

Founder and CEO of M5 Hosting here. We did have a network outage today that affected Hacker News. As with any outage, we will do an RCA and we will learn and improve as a result.<p>I&#x27;m a big fan of HN and YC in general, we host of other YC alum, and I have taken a few things through YC Startup School. During this incident, I spoke to YC personally when they called this morning.

HN is one of the few sites I always keep zoomed-in (around 200%), which led to me finding an interesting bug in Chrome while HN was down: Chrome&#x27;s internal &quot;This site can&#x27;t be reached&quot; page uses the zoom level of the site you would be visiting (if it were up), rather than Chrome&#x27;s default zoom.<p>Screenshot: <a href="https:&#x2F;&#x2F;i.imgur.com&#x2F;VwFtgQh.png" rel="nofollow">https:&#x2F;&#x2F;i.imgur.com&#x2F;VwFtgQh.png</a>

Hacker News is hosted at M5 and they are having a network outage:<p><a href="http:&#x2F;&#x2F;status.m5hosting.com&#x2F;pages&#x2F;incident&#x2F;5407b8e2b002442510000272&#x2F;604fa9a35aef05052e955afe" rel="nofollow">http:&#x2F;&#x2F;status.m5hosting.com&#x2F;pages&#x2F;incident&#x2F;5407b8e2b00244251...</a><p>edit: Unrelated to the Azure outage.

I think I have tried to visit HN for more than 10 times in last 2 hours and failed. This made me realize how much I&#x27;m addicted to HN

Had me quite confused because I&#x27;m also having home internet issues. I was trying to get my laptop to switch to my mobile hotspot and HN is one of the sites I used to test connectivity because a) it&#x27;s almost always available and b) loads very quick.<p>A bit of a mindfuck trying to assess my actual internet connectivity via a site that was also down : )_

The title should be updated to &quot;<i>Productivity was up</i>&quot;.

my fingers automatically just start typing in &quot;news.y&quot; when I&#x27;m idle, I definitely didn&#x27;t know what to do when greeted with a 404!<p>Is there any way to put the HN homepage on an edge cache so at least the homepage shows up? Or am I admitting that I&#x27;m addicted to checking HN too many times a day?

I never expect HN to be down... I asked my wife - hey is the internet down? She said - no, it&#x27;s working for me. I clicked on another site and my mouth dropped.

It says a lot that @hnstatus has not tweeted since 2018.

@dang<p>Would love an updated post on what the current hardware &#x2F; software stack that’s running HN.<p>It’s been years since I’ve seen a post&#x2F;comment on this topic.<p>Are you still running FreeBSD, on a few high frequency cores (iirc)?

Azure AAD also had an outage at this time - perhaps linked in some domino effect, or perhaps a coincidence?<p><a href="https:&#x2F;&#x2F;status.azure.com&#x2F;en-us&#x2F;status" rel="nofollow">https:&#x2F;&#x2F;status.azure.com&#x2F;en-us&#x2F;status</a>

Seeing HN unresolved was a bit weird as it is the best performing site I ever visit on my low bandwidth phone so several times I thought the problem was on my end. But in the end it helped me realize how frequently I dial into HN while it was down. I have a bit of a problem and I think I need to turn on that no procrastination flag on.

I&#x27;ve been on this site 12+ years, and I don&#x27;t ever remember it being down. I assumed we were under nuclear attack.

HN is my homepage and when it wouldn&#x27;t load, I told my coworkers the internet was down and took a 3 hour lunch.

&gt; Back now. Got to write some code for a change... - 5:39<p><a href="https:&#x2F;&#x2F;twitter.com&#x2F;HNStatus&#x2F;status&#x2F;1371576822748487683" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;HNStatus&#x2F;status&#x2F;1371576822748487683</a>

It&#x27;s seems an odd coincidence of this and the Azure AD outage -- I was trying to get to HN to see what people were saying about it!

Something that I learned from this is that HN has a status Twitter. Rarely used though, which is a testament to the team.<p><a href="https:&#x2F;&#x2F;twitter.com&#x2F;HNStatus&#x2F;status&#x2F;1371525940656803848?s=20" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;HNStatus&#x2F;status&#x2F;1371525940656803848?s=20</a>

I thought pg posted &quot;Memphis&quot;.

I didn&#x27;t notice unfortunately due to the Azure outage blowing everything up :(

So strange that this coincided with Azure authentication eating it.

The one time I&#x27;m actually reading HN for actually relevant information for actual work, it&#x27;s down for half a day. Made for a great excuse to take a nap.

I have a massive learning project[1], and I think 2&#x2F;3 of my &quot;to get through as soon as sensible&quot; content is news.ycombinator links.<p>Needless to say, this site is my own personal StackOverflow, and I think there&#x27;s something about ingratitude bouncing around in my mind somewhere.<p>[1]<a href="https:&#x2F;&#x2F;github.com&#x2F;PhilosAccounting&#x2F;ts-learning" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;PhilosAccounting&#x2F;ts-learning</a>

Is HN fully back? Looks like this was a little less than 3 hours total, is that right?

Funny, my first thought was &quot;oh no they&#x27;ve blacklisted VPNs&quot;, can&#x27;t remember when HN was ever down!

It did not seem to affect the Firebase feed.

Curious, what is the uptime of HN - is there some data about that?<p>My guess is around 99.9% ... but maybe that&#x27;s too optimistic?

Who needs so many 9&#x27;s when there&#x27;s actually interesting content we keep coming back for?

We also had issues with our YC application earlier today, was that related to this issue?

It’s OK. The skies haven’t fallen. (And I was able to accomplish something today.)

There was a noticeable increase in productivity during the last hour or so.

It&#x27;s 3pm.... do you know where your servers are?

And now it looks like there is an outage at reddit

Is this related to the big Microsoft outage?

If only they used Kubernetes! &#x2F;s

Since I couldn&#x27;t get to HN, I wrote up how to make the site resilient to outages: <a href="https:&#x2F;&#x2F;gist.github.com&#x2F;peterwwillis&#x2F;ce2bfaba7fc72e4af44c28135ab3db1e" rel="nofollow">https:&#x2F;&#x2F;gist.github.com&#x2F;peterwwillis&#x2F;ce2bfaba7fc72e4af44c281...</a><p>tl;dr 1 server x 2 providers, different regions, replicate content

This site is so reliable, I thought my I.P. had gotten banned.