Since all that really can be deferred from the dumped code is that someone made a typo in a globally included config file (whether it was outside of webroot or not can't really be determined). The obvious issue is that obviously there was no process for testing code prior to deployment. This could have been caught using a standard Dev/Stage/Production environment setup, or even a quick test prior to deployment to Production. Other possible ways to catch this is with proper Unit Testing (which would have broken when not a single service would work due to no config params coming through), or a proper Continuous Integration environment to handle all that.
I'm really beginning to wonder what the hell is going on at Tumblr. Constant downtime to where their down message is the new fail whale, cases of severe downtime with poor communication, and now this.<p>Hopefully the shape up. I'd have serious concerns even running a personal blog on there.
Aren't all bugs effectively a human error at some level? In this case, the human wrote bad PHP.<p>I hope Tumblr will keep us more in the loop on this than when they were down for 24+ hours.
I think they need to further explain what <i>exactly</i> happened than simply stating "human error". If it is avoidable, admit what it is and also let other people to learn this lesson.<p>What configuration error (technically, by definitions from some IEEE guidelines, mistakes) did they make to have this happening?
<i>The fact that this occurred at all is still unacceptable, and we’ll be seriously evaluating and adjusting our processes to ensure an error like this can never happen again.</i><p>I'm not aware of any theorem provers for Apache config files.