It seems like there are a lot of examples of companies handling a security breach or loss of service poorly. Are there examples when a company handled an incident well, especially with a great postmortem writeup?
Edward Tufte's analysis of the Space Shuttle Columbia explosion[0] is by far the most informative post mortem I've seen. It directly impacted everything I've written since reading it.<p>If you hit the link, you'll see the page appears to be a wall of text, not a simple slide or two. As you read deeper into the report, you'll understand that's an intentional aspect of the report. (I'll also note this is the Columbia explosion, not the better known Challenger disaster O-ring post-mortem discussed by Richard Feynman in his autobiography[1], even though that's a great post mortem as well).<p>[0]<a href="https://www.edwardtufte.com/bboard/q-and-a-fetch-msg?msg_id=0001yB" rel="nofollow">https://www.edwardtufte.com/bboard/q-and-a-fetch-msg?msg_id=...</a><p>[1]<a href="https://www.amazon.com/What-Care-Other-People-Think/dp/0393355640/" rel="nofollow">https://www.amazon.com/What-Care-Other-People-Think/dp/03933...</a>
An old colleague of mine worked pretty extensively with a backend engineer on a script to do some data migration on user accounts. The other engineer did most of the work on the script itself. Mind you, the company has millions of users and didn't use staging databases. (No idea if this is still true.)<p>Come D-day, my colleague runs the script on a limited group of users (a 100k or so) to validate. I forget the details, but something in the script was incorrect and ended up breaking some features for all of those users.<p>Once reports started coming in, they were super worried and semi-freaked out. A war room was setup that day and all the people involved jumped in.<p>One of the first things that happened after determining what happened was to calm them down and reassure they weren't in trouble. After that the group worked on a solution for a few hours and established a plan to fix everything.<p>I was actually surprised that the response was so well handled. There was no finger pointing and just a group effort to fix the problem. To me, that's how every problem should be handled, and not instilling fear for losing your job if something bad potentially happens.<p>To anyone who wants to leave replies about staging databases, bad dev practices, etc, don't bother please. This was years ago and it was how things were done at the company. Our team was not part of the backend team or infra and worked with lots of areas of engineering on different issues.
The best postmortem I've seen was actually a conference talk: "Debugging Under Fire: Keep your Head when Systems have Lost their Mind" by Bryan Cantrill (in the GOTO conference, 2017)<p>Here's a YouTube video of it: <a href="https://www.youtube.com/watch?v=30jNsCVLpAE">https://www.youtube.com/watch?v=30jNsCVLpAE</a><p>Here's the slides: <a href="https://gotochgo.com/2017/sessions/86/keynote-debugging-under-fire-keeping-your-head-when-systems-have-lost-their-mind" rel="nofollow">https://gotochgo.com/2017/sessions/86/keynote-debugging-unde...</a><p>It goes into detail about a pretty bad outage (when an entire data center was brought down), the human aspects, automation, how they handled it, the various risks, architectures, how things fail and about software development in general.
Unfortunately I can't share any of ours because they're all proprietary client work, but I think my teams have done a really masterful job. They're among some of the best I've read anywhere. For me the standout traits of a good postmortem are:<p>* Honesty about the reality of the situation; no sugarcoating, no spin<p>* Blameless, factual tone that avoids the passive voice<p>* Describes technical details at a level helpful for practitioners<p>* Makes use of other resources as needed (e.g. references corporate wiki, external ideas, blog posts)<p>* Good writing that's easy to read and is free of grammatical ambiguities and spelling errors
A young guy driving on a country road without a seatbelt rolled his vehicle and died of his head injuries. Quite fascinating to discover that his only real problem was a badly bruised, contused and bleeding brain. Mind you, he probably would have had heart problems in his 50s because his heart arteries already had plaque and he was only in his twenties.<p>That was the best because it was the only postmortem I've seen.
I love premortems at work. Gary Klein came up with the idea of asking why a project could fail before you start it: <a href="https://hbr.org/2007/09/performing-a-project-premortem" rel="nofollow">https://hbr.org/2007/09/performing-a-project-premortem</a>
I learned quite a lot from this <a href="https://blog.cloudflare.com/details-of-the-cloudflare-outage-on-july-2-2019/" rel="nofollow">https://blog.cloudflare.com/details-of-the-cloudflare-outage...</a>
… the description of this topic didn't follow the title in a way that I was expecting at all…<p>I immediately thought of the old GamaSutra (now GameDeveloper.com) postmortems; interviews with members of the teams behind many classic videogames, great late-night reading. <a href="https://www.gamedeveloper.com/audio/10-seminal-game-postmortems-every-developer-should-read" rel="nofollow">https://www.gamedeveloper.com/audio/10-seminal-game-postmort...</a>
I used to have a blog compiling a bunch of them along with articles on best practice creation of post mortems. Unfortunately it never made any money and I took it off line when cpanel put their prices up 1000% and the hosting cost became too much.<p>I still have a back up somewhere and the domain names. Could maybe put it back up one day if I could spare the time and find a very cheap solution. It was a wordpress blog.<p>I realise this doesn't help the OP. I just wanted to vent :-)
There was a really great podcast called The Downtime Project that dissected and discussed a postmortem in each episode. There were like a dozen episodes in the first season and they never did make a second one. Pity, I really, really liked it. Might be up your alley, it's only a couple years out of date now.
We had a payment system for vaccination field workers in africa that stopped working, so people did not get paid. There was a section in the post mortem template that went something like<p>"what is the impact of the error: there is an angry mob with torches demanding to get paid outside"
<a href="https://gvnshtn.com/posts/maersk-me-notpetya/" rel="nofollow">https://gvnshtn.com/posts/maersk-me-notpetya/</a><p>Its a long read, but gives an insight how the ransomware NotPetya crippled Maersk and how they recovered.<p>Looking at how the earlier ransomware WannaCry crippled the crown jewels of many countries, highlights a weakness in non diverse systems.<p>I even know who is behind them, but I cant prove it, so why even mention it? Because I'm getting closer to proving it, which makes this game all the more interesting, even they have weaknesses they have failed to identify!<p>The WannaCry weekend was when I met Dame Stella Rimington and Baron Jonathon Evans hill walking at Scafell Pike and you can call me the world famous Walter Mitty because people are so obedient to authority.