Fundamentals of Incident Management

It&#x27;s not inspiring to me with the &#x27;cool tactical&#x27; framing this article attempts to convey.<p>I&#x27;ve worked as an oncall for a fundamental backbone service of the internet in the past and paged into middle of the night outages. It&#x27;s harrowing and exhausting. Cool names like &#x27;incident commander&#x27; do not change this.<p>We also had a &quot;see ya in the morning&quot; culture. Instead I&#x27;d be much more impressed to have a &quot;see ya in the afternoon, get some sleep&quot; culture.

&gt; which captures all the key log files and status information from the ailing machine.<p>Machine? As in singular machine goes down and you wake up 5 people? That just sounds like bad planning.<p>&gt; Pearson is spinning up a new cloud server, and Rawlings checks the documentation and procedures for migrating websites, getting everything ready to run so that not even a second is wasted.<p>Heroic. But in reality you have already wasted minutes. Why is this not all automated?<p>I understand that this is a simulated scenairo. Maybe the situation was simplified for clarity, but really if a single machine going down leads to this amount of heroics then you should work on those fundamentals. In my opinion.

Nice to see articles like this describing a company&#x27;s incident response process and the positive approach to incident culture via gamedays (disclaimer: I&#x27;m a cofounder at Kintaba[1], an incident management startup).<p>Regarding gamedays specifically: I&#x27;ve found that many company leaders don&#x27;t embrace them because culturally they&#x27;re not really aligned to the idea that incidents and outages aren&#x27;t 100% preventable.<p>It&#x27;s a mistake to think of the incident management muscle as one you&#x27;d like exercised as little as possible when in reality it&#x27;s something that should be in top form because doing so comes with all kinds of downstream values for the company (a positive culture towards resiliency, openness, team building, honesty about technical risk, etc).<p>Sadly this can be a difficult mindset to break out of especially if you come from a company mired in &quot;don&#x27;t tell the exec unless it&#x27;s so bad they&#x27;ll find out themselves anyway.&quot;<p>Relatedly, the desire to drop the incident count to zero discourages recordkeeping of &quot;near-miss&quot; incidents, which generally deserve to have the same learning process (postmortem, followup action items, etc) associated with them as the outcomes of major incidents and game days.<p>Hopefully this outdated attitude continues to die off.<p>If you&#x27;re just getting started with incident response or are interested in the space, I highly recommend:<p>- For basic practices: Google&#x27;s SRE chapters on incident management [2]<p>- For the history of why we prepare for incidents and how we learn from them effectively: Sidney Dekker&#x27;s Field Guide to Understanding Human Error [3]<p>[1] <a href="https:&#x2F;&#x2F;kintaba.com" rel="nofollow">https:&#x2F;&#x2F;kintaba.com</a><p>[2] <a href="https:&#x2F;&#x2F;sre.google&#x2F;sre-book&#x2F;managing-incidents&#x2F;" rel="nofollow">https:&#x2F;&#x2F;sre.google&#x2F;sre-book&#x2F;managing-incidents&#x2F;</a><p>[3] <a href="https:&#x2F;&#x2F;www.amazon.com&#x2F;Field-Guide-Understanding-Human-Error&#x2F;dp&#x2F;1472439058" rel="nofollow">https:&#x2F;&#x2F;www.amazon.com&#x2F;Field-Guide-Understanding-Human-Error...</a>

I&#x27;ve done a LOT of incident management and I&#x27;m not happy about it. The biggest issue I have run into other than burnout is this:<p>Thinking and reasoning under pressure are the enemy. Make as many decisions in advance as possible. Make flowcharts and decision trees with &quot;decision criteria&quot; already written down.<p>If you have to figure something out or make a &quot;decision&quot; then things are really really bad. That happens sometimes, but when teams don&#x27;t prep at all for incident management (pre-determined plans for common classes of problem) every incident is &quot;really really bad&quot;<p>If have a low risk, low cost action with low confidence of high reward, I&#x27;m going to do it and just tell people it happened. Asking means I just lost a half-hour+ worth of money and if I just did it and I was wrong we would have lost 2 minutes of money. When management asks me why I did that, I point at the doc I wrote that my coworkers reviewed and mostly forgot about.<p>A really common example is &quot;it looks like most the errors are in datacenter X&quot;, you fail out of the datacenter. Maybe it was sampling bias or some other issue and it doesn&#x27;t help, maybe the problem follows the traffic, maybe it just suddenly makes things better. No matter what we get signal. Establish well in advance of a situation what the common &quot;solutions&quot; to problems are and if you are oncall and responding, then just DO them and document+communicate as you do.

It sort of baffles me how much engineer time is seemingly spent here designing and running these &quot;gamedays&quot; vs just improving and automating the underlying systems. Don&#x27;t glorify getting paged, glorify systems that can automatically heal themselves.<p>I spend a good amount of time doing incident management and reliability work.<p>Red team&#x2F;blue team gamedays seems like a waste of time. Either you are so early on your reliability journey that trivial things like &quot;does my database failover&quot; are interesting things to test (in which case just fix it). Or, you&#x27;re a more experienced team and there&#x27;s little low hanging reliability fruit left. In the later, gamedays seem unlikely to that closely mimic a real world incident. Since low hanging fruit is gone, all your serious incidents tend to be complex failure interactions between various system components. To resolve them quickly, you simply want all the people with deep context on those systems quickly coming up with and testing out competing hypotheses on what might be wrong. Incident management only really matters in the sense that you want to allow the people with the most system context to focus on fixing the actual system. Serious incident management really only comes into play when the issue is large enough to threaten the company + require coordinated work from many orgs&#x2F;teams.<p>My team and I spend most of time thinking about how we can automate any repetitive tasks or failover. In the case something can&#x27;t be automated, we think about how we can increase the observability of the system, so that future issues can be resolved faster.

There is a month and day, Feb 15, in the header, but no year. I can&#x27;t figure out if that&#x27;s ironic or apropos, since this story reads like a thriller from perhaps ten years ago, but the post date appears to have been 2020-02-15 - yikes.

I may never understand why some places are all about assigning titles and roles in this kind of thing. You need one, maybe two, plus a whole whack of technical skills from everyone else.<p>Also, conference calls are death.

So they are testing against fully awake people at 2:30pm and expecting similiar results at 4:30am after heavy drinking.

It doesn&#x27;t match my experience, with a real incident.<p>I was a dev in a small web company (10 staff), moonlighting as sysadmin. Our webserver had 40 sites on it. It was hit by a not-very-clever zero-day exploit, and most of the websites were now running the attacker&#x27;s scripts.<p>It fell to me to sort it out - the rest of the crew were to keep on coding websites. The ISP had cut off the server&#x27;s outbound email, because it was spewing spam. So I spent about an hour trying to find the malicious scripts, before I realised that I could never be certain that I&#x27;d found them all.<p>You get an impulse to panic when you realise that the company&#x27;s future (and your job) depends on you not screwing up; and you&#x27;re facing a problem you&#x27;ve never faced before.<p>So I commissioned a new machine, and configured it. I started moving sites across from the old machine to the new one. After about three sites, I decided to script the moving work. Cool.<p>But the sites weren&#x27;t all the same - some were Drupal (different versions), some were Wordpress, some were custom PHP. It worked for about 30 of the sites, with a lot of per-site manual tinkering.<p>Note that for the most part, the sites weren&#x27;t under revision control - there were backups in zip files, from various dates, for some of the sites. And I&#x27;d never worked on most of those sites, each of which had its own quirks. So I spent the next week making every site deploy correctly from the RCS.<p>I then spent about a week getting this automated, so that in a future incident we could get running again quickly. Happily we had a generously-configured Xen server, and I could test the process on VMs.<p>My colleagues weren&#x27;t allowed to help out, they were supposed to go on making websites. And I got resistance from my boss, demanding status updates (&quot;are we there yet?&quot;)<p>The happy outcome is that that work became the kernel of a proper CI pipeline, and provoked a fairly deep change in the way the company worked. And by the end, I knew all about every site the company hosted.<p>We were just a web-shop; most web-shops are (or were) like this. If I was doing routine sysadmin, instead of coding websites, I was watched like a hawk to make sure I wasn&#x27;t doing anything &#x27;unnecessary&#x27;.<p>This incident gave me the authority to do the sysadmin job properly; and in fact it saved me a lot of sysadmin time - because previously, if a dev wanted a new version of a site deployed, I had to interrupt whatever I was doing to deploy it. With the CI pipeline, provided the site had passed some testing and review stage, it could be deployed to production by the dev himself.<p>It would have been cool to be able to do recovery drills, rotating roles and so on; but it was enough for my bosses that more than one person knew how to rebuild the server from scratch, and that it could be done in 30 minutes.<p>Life in a small web-shop could get exciting, occasionally.