Post-Mortem for Google Compute Engine’s Global Outage on April 11

This is a very good Post-Mortem.<p>As I assumed it was kind of a corner case bug meet corner case bug met corner case bug.<p>This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them?<p>Making software is hard....

&gt; There are a number of lessons to be learned from this event -- for example, that the safeguard of a progressive rollout can be undone by a system designed to mask partial failures -- ...<p>This is a really important point that should be more generally known. To quote Google&#x27;s own &quot;Paxos Made Live&quot; paper, from 2007:<p>&gt; In closing we point out a challenge that we faced in testing our system for which we have no systematic solution. By their very nature, fault-tolerant systems try to mask problems. Thus they can mask bugs or configuration problems while insidiously lowering their own fault-tolerance.<p>As developers we can try to bear this principle in mind, but as Monday&#x27;s incident demonstrated, mistakes can still happen. So, has anyone managed to make progress toward a &quot;systematic solution&quot; in the last 9 years?

Great writeup! PS: &quot;To make error is human. To propagate error to all server in automatic way is devops.&quot; -DevOps Borat

It looks like there were at least three catastrophic bugs present:<p>1. Evaluated a configuration change before the change had finished syncing across all configuration files, resulting in rejecting the change.<p>2. So it tried to reject the change, but actually just deleted everything instead.<p>3. Something was supposed to catch changes that break everything, and it detected that everything was broken but its attempt to do anything to fix it failed.<p>It is hard to imagine that this system has good test coverage.

<i>In this event, the canary step correctly identified that the new configuration was unsafe. Crucially however, a second software bug in the management software did not propagate the canary step’s conclusion back to the push process, and thus the push system concluded that the new configuration was valid and began its progressive rollout.</i><p>Classic Two Generals. &quot;No news is good news,&quot; generally isn&#x27;t a good design philosophy for systems designed to detect trouble. How do we know that stealthy ninjas haven&#x27;t assassinated our sentries? Well, we haven&#x27;t heard anything wrong...

I&#x27;m waiting for the time when they push over the air updates to airplanes in flight.<p>&quot;You can fly safely, we have canaries and staged deployment&quot;<p>A year forward:<p>&quot;Unfortunately because the canary verification as well as the staged deployment code was broken, instead of one crash and 300 dead, an update was pushed to all aircraft, which subsequently caused them to crash, killing 70,000 people.&quot;<p>I&#x27;m not 100% sure why they don&#x27;t do the staged deployment for google scale server networking over a few days (or even weeks in some cases) instead of a few hours, but I don&#x27;t know the details here...<p>It&#x27;s good that they had manually triggerable configuration rollback possibility and a pre-set policy so it was solved so quickly.

At Google, they do these really awesome post-mortems when there&#x27;s a major failure. It provides a point of reflection, and are usually well written entertaining reads. Didn&#x27;t know they made (some?) public.<p>They&#x27;re a good learning exercise writing one, and is more of a learning exercise than a punishment.

Completely off topic, but this thread is an example of why I (and a lot of people) want collapsible comments native to HN. I&#x27;m on my phone, in Safari, and I had to scroll for over 20 seconds just to reach the second comment. The first comment was a tangent about self-driving cars, which while relevant, I didn&#x27;t want to read about.

&gt;However, in this instance a previously-unseen software bug was triggered, and instead of retaining the previous known good configuration, the management software instead removed all GCE IP blocks from the new configuration and began to push this new, incomplete configuration to the network.<p>&gt;Crucially however, a second software bug in the management software did not propagate the canary step’s conclusion back to the push process, and thus the push system concluded that the new configuration was valid and began its progressive rollout.<p>I assume the software was originally tested to make sure it works in case of failure. It would be interesting to know exactly what the bug was and why it didn&#x27;t show in tests.

Attention startups: <i>this</i> is what incident post-mortems should look like.

This is very interesting. From the little I understand (sorry for using AWS terms as I am more versed with AWS than GCE) this can happen to AWS as well right? even if your software is deployed to multiple AZs &#x2F; multiple regions, if bad routing &#x2F; network configuration makes it through the various protection mechanisms then basically no amount of redundancy can help if your service is part of the non functional IP block. I mean it seems no matter how redundant you are, there will always be somewhere along the line a single point of failure, even if it has multiple mechanism to prevent it from happening, if all of these mechanisms fail, then it&#x27;s still a single point. What prevents this from happening at Azure &#x2F; AWS? Is there anything that general internet routing protocols need to change to prevent it from happening?<p>e.g. I&#x27;m sure that we will never hear that Bank of X has transferred a billion dollar to an account but because of propagation errors it published only the credit but didn&#x27;t finish the debit and now we have two billionaires. This two or more phase commit is pretty much bulletproof in banking as far as I know, and banks are not known to be technologically more advanced than Google, how come internet routing is so prone to errors that can an entire cloud service unavailable for even a small period of time? I&#x27;m far from knowing much about networking (although I took some graduate networking courses, I still feel I know practically nothing about it...) So I would appreciate if someone versed in this ELI5 whether it can happen in AWS and Azure regardless of how redundant you are, (which leads to a notion of cross cloud provider redundancy which I&#x27;m sure is used in some places) and whether the banking analogy is fair and relevant, and if there are any RFCs to make world-blackout routing nightmares less likely to happen.

&gt; . Internal monitors generated dozens of alerts in the seconds after the traffic loss became visible at 19:08 ... revert the most recent configuration changes ... the time from detection to decision to revert to the end of the outage was thus just 18 minutes.<p>It&#x27;s certainly good that they detected it as fast as they did. But I wonder if the fix time could be improved upon? Was the majority of that time spent discussing the corrective action to be taken? Or does it take that much time to replicate the fix?

&gt; Finally, to underscore how seriously we are taking this event, we are offering GCE and VPN service credits to all impacted GCP applications equal to (respectively) 10% and 25% of their monthly charges for GCE and VPN.<p>These credits exceed what is promised by Google Cloud in their SLA&#x27;s for Compute Engine and VPN service!

Nice post mortem.<p>That outtage gives GCE at best a four 9&#x27;s reliability for 2016.

I always like Google&#x27;s serious attitude towards engineering, even after they have made some mistakes, they never try to hide anything.

&gt; However, in this instance a previously-unseen software bug was triggered, and instead of retaining the previous known good configuration, the management software instead removed all GCE IP blocks from the new configuration<p>&gt; Crucially however, a second software bug in the management software did not propagate the canary step’s conclusion back to the push process<p>I&#x27;m sure the devil is in the details, but generally speaking, these are 2 instances of critical code that gets exercised infrequently, which is a good place for bugs to hide.

Do SLAs even matter in the slightest? Or are they just sort of &quot;feel-good&quot; things or ways for negotiators to demonstrate their worth?

&quot;Lessons learned from reading post-mortems&quot; <a href="http:&#x2F;&#x2F;danluu.com&#x2F;postmortem-lessons&#x2F;" rel="nofollow">http:&#x2F;&#x2F;danluu.com&#x2F;postmortem-lessons&#x2F;</a> is a good place to dig deeper<p>The first graph quoted from a survey paper is a classic fitting the GCE outage well:<p>Initial error --92%--&gt; Incorrect handling of errors explicitly signaled in software

I love his signature: &quot;Benjamin Treynor Sloss | VP 24x7&quot;.

&gt; However, in this instance a previously-unseen software bug was triggered, and instead of retaining the previous known good configuration, the management software instead removed all GCE IP blocks from the new configuration and began to push this new, incomplete configuration to the network.<p>Always test your crash &#x2F; exception handling &#x2F; special case termination+recovery code in production.<p>I have seen this too often. Most often in in &quot;every day&quot; cases when service has a &quot;nice&quot; catch way of stopping and recovering. Then has a separate &quot;if killed by SIGKILL&#x2F;immediate power failure&quot; crash and recovery. This last bit never gets tests and run in production.<p>One day power failure happens, service restart and tries to recover. Code that almost never runs, now runs and the whole thing goes into an unknown broken state.

This isn&#x27;t the first time a config system at Google causes a major outage.<p><a href="https:&#x2F;&#x2F;googleblog.blogspot.com&#x2F;2014&#x2F;01&#x2F;todays-outage-for-several-google.html" rel="nofollow">https:&#x2F;&#x2F;googleblog.blogspot.com&#x2F;2014&#x2F;01&#x2F;todays-outage-for-se...</a>

My post yesterday seems even more relevant today: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=11477552" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=11477552</a><p>It&#x27;s a shame it&#x27;s not easier or more common for people to create clones of (most|all) of their infrastructure for testing purposes.<p>Something like half of outages are caused by configuration oopsies.<p>If you accept that configuration <i>is</i> code, then you also come to the following disturbing conclusion: the usual test environment for critical network-related code in most environments is the production environment.

For the amount this cost them, they should have bought CloudFlare. If you play with [global BGP anycast] you are bound to get burned. This is not the first time that BGP took out your entire routing. This is probably not the last time that BPG will take out your entire routing. Whoever&#x27;s job it was to watch the routing, I am sorry.<p>Pulling your own worldwide routes because you have too much automation; it will make a good story once it&#x27;s filtered down a bit! Icarus was barely up in the air, too early for a fall.

The thing that stood out for me was:<p>&quot;...team...worked in shifts overnight...&quot;

How important for redundancy&#x2F;quality of service is the feature of advertising each region&#x27;s IP blocks from multiple points in Google&#x27;s network? It seems like region isolation is the most important quality that Google&#x27;s network could provide, and their current design is what made something like this possible, not just the bugs in the configuration propagation. They mention the ability of the internet to route around failures, so why not rely on that instead?

as devops Borat was saying all along, automated propagation of a error as the main root cause here. A error (new configuration) should be rolled out site by site - ok us-east1, move onto us-west1 ... ok, move onto ... . A canary site may be the first in sequence, yet success (&quot;no failure reported&quot;) can&#x27;t be a big &quot;ok&quot; for automated push to all sites at the same time.

I hope that one of their solutions is the obvious one; make change control testing a closed loop instead of an open loop. (Watch for &#x2F;success&#x2F; reported instead of failure notification.)

&gt; configuration file<p>configuration files strike again - remember knight capital?

What does Google use for BGP? Quagga, OpenBGPD, BIRD, their own?<p>Also, does anyone have a link to statistics on global BGP software usage? I&#x27;m curious what the marketshare looks like.

The postmortem used the word &quot;quirk.&quot; They might consider drilling down on the specifics there. Especially if that is the heart of the bug&#x2F;accident.

Just a thought. Maybe change the name from &#x27;post-mortem&#x27; to, anything else before the event actually is a post-mortem.

Networking issues in either the storage or communication subsystems of any platform normally result in wide-spread disruptions.

What is the reason different GCE regions use the same IP blocks?

What is defense in depth? It is mentioned as a core principle.

chaos monkey?

&gt; Crucially however, a second software bug in the management software did not propagate the canary step’s conclusion back to the push process, and thus the push system concluded that the new configuration was valid and began its progressive rollout.<p>Perhaps the progressive rollout should wait for an affirmative conclusion instead of assuming no news is good news? I&#x27;m not being snarky, there may be some reason they don&#x27;t do this.

TLDR; they simply didn&#x27;t test their (global!) custom route announcement management software. An edge case was triggered in production, and they gee-whiz-automatically went offline. Epic fail.<p>PS. To the downvoters, truth hurts.

A bit of topic, but it really bugs me, that the banner on the top so pixilated.

DRY &quot;The inconsistency was triggered by a timing quirk in the IP block removal - the IP block had been removed from one configuration file, but this change had not yet propagated to a second configuration file also used in network configuration management.&quot;

I think most people are missing the main failure point: Why does one change propagate automatically to all regions?<p>All this could have been contained if they deployed changes on different regions at different times. That would also help with screwing less your overseas users by running a maintenance at 10am their local time :-)