This is essentially the definition of survivorship bias.<p><a href="http://www.fastcodesign.com/1671172/how-a-story-from-world-war-ii-shapes-facebook-today" rel="nofollow">http://www.fastcodesign.com/1671172/how-a-story-from-world-w...</a><p>"In WWII, Allied bombers were key to strategic attacks, yet these lumbering giants were constantly shot down over enemy territory. The planes needed more armor, but armor is heavy. So extra plating could only go where the planes were being shot the most."...<p>"[A guy in charge] said the military didn’t need to reinforce the spots that had bullet holes. They needed to reinforce the spots that didn’t have bullet holes."<p>Having experienced 2 "WFIO" things in the last 6 months, I agree that entrepreneurs need to be resilient, but I think it's more valuable to read post mortems than these "we almost died" posts because they prove more instructional.<p>For instance, Aaron's post about Tutorspree is a really useful post for me, although it definitely has less of a feel good vibe to it.<p><a href="http://www.aaronkharris.com/when-seo-fails-single-channel-dependency-and-the-end-of-tutorspree" rel="nofollow">http://www.aaronkharris.com/when-seo-fails-single-channel-de...</a>
<i>> Get all the brains around the table</i><p>An important addition to this: get everyone involved, but also listen to their ideas.<p>Having been a part of a sinking ship, the most frustrating thing about having everyone pour their blood, sweat, and tears into reviving the company is when the top brass decides to ignore all of the hard work and carry on with their own ideas that were never cleared with anyone else. They were their own iceberg.
Fresh out of college I was responsible for a WFIO at a 40 person company - ultimately had to leave.<p><a href="http://edu.mkrecny.com/thoughts/how-i-fired-myself" rel="nofollow">http://edu.mkrecny.com/thoughts/how-i-fired-myself</a><p>"Um those [backups] got really expensive, so we stopped doing them about a month ago" sounds painfully familiar.
"Nothing is beneath a leader in times of crisis"<p>I've worked for people that believe this, and others that don't. Nothing builds morale and loyalty like rolling up your sleeves and doing the hard stuff with your employees.<p>I once had a project manager who couldn't write a lick of code, but whenever a deadline approached or something blew up, he'd be right there at 10pm with everyone else bringing in fresh pizza, testing whatever he could, and giving pep talks. Contrast this with someone that says, "get this done by tomorrow, OK?" and then walks out the door. Who do you think is going to have a higher performing team in the long run?
I had serious WFIO case a few years ago. We were building a crypto solution on J2ME phones.<p>The solution was days away from roll-out with our first big corporate client. Late on a Friday afternoon we were busy with final field testing - paying students a few $ to use their phones to test the app.<p>As the test data came in we realised we had a major problem: a small % of the phones weren't returning the correct test vectors for hashing algo.<p>After checking for obvious user error we came to the conclusion that something big was broken. Specific firmware sets didn't execute the crypto part of the code correctly. The entire value-proposition was that it works on every phone that can run an app, so it was a pretty big deal. I thought we were totally fucked.<p>We didn't sleep for two days and finally found the bug in the way the phones implement a bit-shift operation (doesn't carry a bit about 1/10000000 times). Then had to figure out a workaround that was still fast enough.<p>We shipped a fixed version before the end of the weekend, but I wouldn't wish that kind of stress on anyone.
I don't understand the meaning of "crack the egg with a sledgehammer." I would think it means "to use way more force than is necessary," but the context is more along the lines of "get the problem solved at all costs."<p>I also followed the link to the other article where that phrase is used[1], and found something a bit concerning: "Nawaf moved the entire engineering team over to work on it. He called them all in to work nights and weekends until it was fixed, [...] Nawaf saved our bacon."<p>Umm, what about those engineers? I'll give the benefit of the doubt that they were rewarded appropriately, but the wording here seems to almost deliberately stoke the developers vs. management flames, especially coming from a VC...<p>[1] <a href="http://scott.a16z.com/2014/02/03/harvey-keitel-ceo/" rel="nofollow">http://scott.a16z.com/2014/02/03/harvey-keitel-ceo/</a>
Am I the only one that thought the query string was actually more interesting than the article?<p>?utm_content=bufferc2fec&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer<p>I'm glad that ycombinator isn't listed as the source, at least. I think it would be fairly interesting to see what kind of data they have about various social medias and link propagation.<p>And also, a slight sickening sensation that marketers are doing this. Not that I'm surprised, it's just that I'm sure they have papers written saying that "Twitter has a 64% link click rate if you write your post like X, whereas Facebook can achieve as high as 70% if you do Y."<p>I know it has been going on for years on the web, and decades for other marketing, but here it's plain to see. It fills me with unease, like I'm no longer in control and what I like in life is already planned out.
"It's never really as bad as it seems."<p>Very true. Many times when something happens the automatic response is to focus on the worst case scenario instead of the immediate problem at hand.<p>Also, I couldn't select any text to copy and paste.
The most interesting take away from that was how resilient companies appear to be despite their leaders best efforts to sabotage everything. I think the real lesson is (and I don't like it): if you want your company to survive you need to be good at spin.<p>In the case of hotmail, how did he ever let a situation arise where one bug in a nightly cron script could obliterate his users emails?! No backups because they cost too much?? "the sun, the moon and the stars lined up against us for it to happen" No, you were just reckless and too busy chasing growth.<p>With Ironport, he says his whole business was built on the back of anti-spam yet they were totally unprepared when their partner who provided the anti-spam tech they used pulled the rug from under them (despite knowing full well this day would come). The real lesson here should be: try not to outsource the key component upon which your entire business depends.<p>I had to laugh at the end when he says the only time he's had a failure was when he was using his own limited resources to fund a business and not investor money. Well yes... mistakes are easier to hide/absorb when you're swimming in cash. Not so much when things are tight and you can't afford to learn after the fuck up.
Both Hotmail & Ironport reacted with a customer-centric focus (e.g. Hotmail: "How do we recover their emails? What else can we provide customers in the future? What steps will we take so this never happens again?")<p>I would guess a lot of companies in a WFIO situation that take a CYA approach rather than customer-first face a worse fate.
Question - why did IronPort cancel the contract with their customers if they just got the feature implemented that they seemed to want? Was it a done deal - i.e. they were losing those customers anyways?<p>Also, survivorship bias much? Two examples --> conclusion? Was nice hearing about those company's histories though, didn't know about the Hotmail thing.
The same principles were true and applicable in a much larger near-implosion that I experienced from 2008-2012 working for a "too big to fail" financial company. Strong, out-front leadership, a strong team led the way. It took time for us to realize that it wasn't as bad as it seemed. We had to learn that the external pressure from the media, public and even friends and family was only a distraction.<p>In this situation it was quickly apparent that my role was to focus on a solution, not dwell on the problem or the cause. This should be true for everyone on the team, unless you are the specific individual who caused a global financial meltdown, deleted all the email or caused a critical failure.
I really admire Scott from what he's written. He can be an inch close and still get out of WFIO situation. And he seems to know the difference when it's doomed like with e-commerce business. So keep rowing, Scott!
I am disappointed to see the top comment here criticizes this as "survivorship bias." Maybe the survivors have something useful to communicate that can help others survive what looks "impossible" when you first run into it? Maybe that's the point?<p>Though this probably shouldn't be exactly a surprise to me given how much shit people give me any time I try to talk about getting well after doctor's basically wrote me off for dead in some sense. No one wants to learn from that either and I honest to god don't get it.
Great points, especially leading from the front when times are tough. I didn't think Hotmail would have survived the outages, but they came out just fine.
Nice article.<p>I particularly agree with the point on leadership. If you can motivate those around you, people perform so much better and can sometimes come up with better solutions. Doesn't always guarantee you coming back from the brink, but stands you in good stead.
"We're Fucked, It's Over" strikes me as a pretty dramatic response to a technology vendor failing to renew a contract. Unless they have extra-special access to customers or customer data, whatever they do can be replicated in-house.
Fascinating to read a 1st person account of a near-death moment. "Companies are damn resilient" - I think entrepreneurs are damn resilient, too, for making it through crises like this over & over...
My palms got sweaty just reading the hotmail portion. I think someone could make a really great horror book that is just a collection of stories like this. I know it would give me nightmares.
if it's never as bad as it seems: does this make a case for optimism or realism on a team?<p>regardless, that's precisely why you need a cofounder who balances you. whether it's a different perspective, questioning some of your choices, or holding you accountable (and holding you up): you need it. you'll make an order of magnitude more poor choices on your own.
<i>The leader needs to be the first one there, the last one to leave, and be willing to do anything it takes – like answer customer care calls or personally drive a replacement part to an irate customer. Nothing is beneath a leader in times of crisis.</i><p>I like this, because it says two things at once, both true.<p>First, if you're in a position of ownership/authority but you're not willing or able to do the grunt work, you won't have the credibility for long. People will follow orders out of fear of getting fired, but you'll never get more than the bare minimum. There's a point where leaders are replaced or outnumbered by true executives (lazy, bikeshedding, rent-seeking parasites) and after that point, the organization can't even motivate shit to stink.<p>Second, it might be that <i>it's just not worth it</i> to lead from the front. That means that you're not <i>really</i> in a position of leadership. You might be a middle manager who realizes that the people above you will never buy in to what you're doing. Then you'll probably lose that desire to make those sacrifices. That's fine. You shouldn't tie yourself to the mast, at that point, because you're not really in a leadership role anyway.