Final Root Cause Analysis of Nov 18 Azure Service Interruption

Nice writeup. I hope that the engineer in question didn&#x27;t get fired or anything. One of the challenges in SRE&#x2F;Ops type organizations is to be responsible, take ownership, put in the extra time to fix things you break, <i>but keep the nerve to push out changes.</i> Once an ops team loses its willingness to push large changes, the infrastructure calcifies and you have a much bigger problem on your hand.

The really big missing piece that I found in this post mortem is if it only took 30 minutes to revert the original change, why did it take over ten hours to restart the Azure Blob storage servers? This was neatly elided in the last sentence of this paragraph of their writeup:<p>&quot;.... We reverted the change globally within 30 minutes of the start of the issue which protected many Azure Blob storage Front-Ends from experiencing the issue. The Azure Blob storage Front-Ends which already entered the infinite loop were unable to accept any configuration changes due to the infinite loop. These required a restart after reverting the configuration change, extending the time to recover.&quot;<p>The ten+ hours extension was the vast majority of the outage time; why wasn&#x27;t the reason for this given? More importantly, what will be done to prevent a similar extension in the time Azure spends belly up if at some point in the future, the Blob servers go insane and have to be restarted?

<i>&quot;These Virtual Machines were recreated by repeating the VM provisioning step. Linux Virtual Machines were not affected.&quot;</i><p>So Azure supports Linux VMs?! Microsoft does so little Azure advertising that I had to learn this fact from their RCA. Apparently they do support it since 2012: <a href="http://www.techrepublic.com/blog/linux-and-open-source/microsoft-now-offering-linux-on-azure-what-does-this-mean/" rel="nofollow">http:&#x2F;&#x2F;www.techrepublic.com&#x2F;blog&#x2F;linux-and-open-source&#x2F;micro...</a> but it is likely that many non-users of Azure do not know this.

&gt; The engineer fixing the Azure Table storage performance issue believed that because the change had already been flighted on a portion of the production infrastructure for several weeks, enabling this across the infrastructure was low risk.<p>Ugh, I wouldn&#x27;t want to be that guy (even if there would be no direct repercussions). That said, and as others have highlighted - kudos on the writeup and openness.

Shit happens and at this scale it happens big. I wish everyone would provide details like this when the fan gets hit or your security fails. I&#x27;m glad I never have to deal with scale like this, it&#x27;s pretty scary.

Impressive non-jargonized report. I would have &quot;quantified&quot; the &quot;small number&quot; but kudos anyway to Microsoft for taking this path towards transparency.

This is a good effort. I do have some concerns about it.<p>A true root cause would go deeper and ask why is it that an engineer could solely decide to roll out to all slices ?<p>The surface-level answer is that Azure platform lacked tooling. Is that the cause or an effect ? I think it is an effect. There are deeper root causes.<p>Let&#x27;s ask -- why was it that the design allowed one engineer to effectively bring down Azure ?<p>We often stop at these RCAs when it gets uncomfortable and it starts to point upwards.<p>I say this to the engineer who pressed the buttons: Bravo! You did something that exposed a massive hole in Azure which may have very well prevented a much bigger embarrassment.

I&#x27;m pretty impressed with the openness of this statement.

&quot;Unfortunately, the configuration tooling did not have adequate enforcement of this policy of incrementally deploying the change across the infrastructure.&quot; They relied on tooling to do the review of the last step of the process? I would have thought there were a few layers of approval that goes along with that final push into mission critical infrastructure.

Pros: -they are sharing info -they allowed some caustic comments to remain at the bottom of the page (so far).<p>Cons: -This is almost 30 days after the incident -Look at the regions, it was global! -This was a whole chain of issues. I count it as 5 separate issues. This goes deep into how they operate and it does not paint a picture of operational maturity:<p>1: configuration change for the Blob Front-Ends exposed a bug in the Blob Front-Ends<p>2: Blob Front-Ends infinite loop delayed the fix (I count this as a separate issue though I expect some may not)<p>3: As part of a plan to improve performance of the Azure Storage Service, the decision was made to push the configuration change to the entire production service<p>4: Update was made across most regions in a short period of time due to operational error<p>5: Azure infrastructure issue that impacted our ability to provide timely updates via the Service Health Dashboard<p>That is quite a list. [Edit : formatting only]

Why are they so quiet about SLA credit? Not a word for a month and for a year I have been wasting good money on doubling up services to be inside the SLA + also deploying cross region to ensure zero downtime, what a joke. Surely Azure are not hoping we will forget?

TL;DR<p>&gt;In summary, Microsoft Azure had clear operating guidelines but there was a gap in the deployment tooling that relied on human decisions and protocol.

<i>After analysis was complete, we released an update to our deployment system tooling to enforce compliance to the above testing and flighting policies for standard updates, whether code or configuration.</i><p>Hopefully there is a way to disable this policy adherence for when you really need to push out a configuration or code change everywhere quickly.

I cannot believe how many times I have seen a PROD (or new env X) deployment go bad from configuration issues. At least they separate configuration deployments from code deployments, that&#x27;s a good sign. Why not take it a step <i>further</i> and instead of doing config deployments, use a config server?

If utility computing is to is be taken seriously, then it has to institute the same kind of discipline that we see occurring in the airline industry. Recent examples come to mind: pilot letting songstress hold the wheels and wearing the pilots cap - fired. Airline executive overruling pilot over macadamia nuts - million dollar fine.<p>If we wish for a future where cloud computing will be considered reliable enough for air traffic control systems, then management of these infrastructure requires a level of dedication and commitment to process and training.<p>Failover zones need to be isolated not only physically, but also from command and control. A lone engineer should not have sufficient authority or capability to operationally control more than one zone. It is extremely unnerving for enterprises to see that a significant infrastructure like Azure has a root account which can take down the whole of Azure.

I hope the engineer in question did not get fired.<p>I also hope that no one who recommended Azure to their employer got fired either.

Only one question: Will the engineer be fired?

OSS is alive and well on the Azure Platform www.microsoft.com&#x2F;openness

Does anyone else see the missing piece to this post mortem? An infinite loop made its way onto a majority(? all?) of production servers, and the immediate response is more or less &#x27;we shouldn&#x27;t have deployed to as many customers, failure should have only happened to a small subset&#x27;?<p>I agree that improvements made to their deployment tooling were good and necessary, take the human temptation to skip steps out of the equation.<p>But this exemplifies a <i>major</i> problem our industry suffers from, in that it just taken as a given that critical errors will sometimes make their way into production servers and the best we can do is reduce the impact.<p>I find this absolutely unacceptable. How about we short circuit the process and identify ways to stop that from happening? Were there enough code reviews? Did automated testing fail here? Yes I&#x27;m familiar with the halting problem and limitations of formal verification on turing complete languages, but I don&#x27;t believe it&#x27;s an excuse.<p>This is tantamount to saying &quot;yeah sometimes our airplanes crash, so from now on we&#x27;ll just make sure we have less passengers ride in the newer models&quot;.