Knightmare: A DevOps Cautionary Tale (2014)

Amusing personal anecdote -- the Knight debacle caused the market in general to tumble. The week before a coworker of mine -- sure of a market drop but for other reasons -- had bought a raft or puts on the S&P 500. When I saw looking glum at work after the Knight news broke, I asked him what was wrong didn't you make a ton? Yeah, he said, but I can't get out cause my account's with Knight.

Back when I used to smoke I would ocassionally hang out with this guy from an investment bank that traded on the Japanese exchange. They had really cool working hours (started a lot later in the day) because we were based in Hawaii which is a few hours behind Japan.<p>Anyway, the guy told me that they had multiple big red physical kill switches so that they could immediately turn things off if shit ever hit the fan with their systems.<p>If you have ever spent time in Michigan you&#x27;ll notice that the manufacturer test vehicles have a big ass red button on the dashboard to kill the vehicle in case something goes wrong.<p>I cannot imagine doing anything remotely close to this sort of thing without a big ass red kill switch on my desk.

I&#x27;m not sure the conclusion of the post is the &quot;One and Only Answer&quot; because a fully automated deploy process has another risk that has bitten both AWS and Google at some point: fully automatically taking down huge amounts of instances.

Previous discussions on HN:<p>2014: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=7652036" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=7652036</a><p>2015: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=8994701" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=8994701</a>

This is less DevOps and more poor software engineering practices (code reviews, unit testing, paying off your technical debt through refactoring&#x2F;removing old code, etc), although properly managing and instrumenting deploys might have stemmed the bleeding and kept losses manageable.<p>It&#x27;s good though; poor decisions must have a cost. The only way to enforce good engineering practices that are human time intensive is for there to be a cost not to.

off topic, but a &quot;knightmare&quot; is also a chess term. It&#x27;s a good-knight-vs-bad-bishop position that went horribly wrong for the owner of the bishop.

What&#x27;s to take away from this?<p>Automate deployment? Fine but boring. That&#x27;s the prevailing dogma today. I don&#x27;t remember where the devops hype train was in 2012. Package management had already been a solved problem for years even though it was (and continues to) be regarded as involving too much &quot;icky reading&quot; and a repository system using plain directories on vanilla webservers; all way too unoptimized for resume padding.<p>Learn how to identify and manage risk like an engineer? Understand how business process and software can implement risk controls and mitigations?<p>I kid, so I don&#x27;t cry.

(2014)