Maybe folks will start to realize that an EDG (Emergency Diesel Generator) without critical support systems is useless.<p>I was just reading about the Fukushima accident, and how most of the EDG's failed because they were water-cooled, and while the EDG's were located out of harms way, the pumps for providing cooling water were located on low ground and were damaged by the tsunami. <a href="http://fukushima.ans.org/report/accident-analysis" rel="nofollow">http://fukushima.ans.org/report/accident-analysis</a><p>Yesterday I saw a story about the Datagram data center in NYC having to shut down. From their reports about the damage (<a href="http://www.datagram.com" rel="nofollow">http://www.datagram.com</a>):<p>"As of 5pm on October 29, 2012, Datagram had thoroughly tested its emergency systems at 33 Whitehall, NYC fully staffed and awaiting the storm to hit Manhattan's shores. Once ConEd lost power to Lower Manhattan, Datagram's emergency systems kicked on maintaining power to Datagram's datacenter. Unfortunately, within a couple hours of the storm hitting Manhattan's shores, the building's entire basement, which houses the building's fuel tank pumps and sump pumps, was completely filled with water and a few feet into the lobby. Due to electrical systems being underwater the building was forced to shut down to avoid fire and permanent damage."<p>It's pretty obvious that, despite all the disaster planning done in the past, Datagram (and TEPCO, and others) have really neglected to appreciate the potential modes of failure for their backup power systems. In both cases, they misunderstood the threat to critical backup power infrastructure. If your EDG is on the roof but the fuel pumps and electrical switchgear is in the basement, what will happen during a flood? If your EDG's are on high ground but the pumps to cool them are not, what happens during a tsunami?
Here is some information on the setup @ 75 Broad St:<p>"The 17th and 18th floors of 75 Broad have been reserved for generator farms that can accommodate as many as 40 machines. Big doors will be installed in the facade of both floors so the generators can be rigged into the building.<p>A 41,000-gallon fuel tank is being installed in the basement, with a separate generator and three redundant pumps to supply the generators on the 17th and 18th floors. Each tenant will own its own generator -- E-Spire already has one installed outside on the setback on the 17th floor -- but the building will sell them fuel."<p><a href="http://www.nytimes.com/1999/10/10/realestate/commercial-property-75-broad-street-turning-buildings-into-telecommunications.html?pagewanted=all&src=pm" rel="nofollow">http://www.nytimes.com/1999/10/10/realestate/commercial-prop...</a>
Squarespace is one of the companies in that data center. See here for their story (and some photos):<p><a href="http://blog.squarespace.com/" rel="nofollow">http://blog.squarespace.com/</a>
<a href="http://status.squarespace.com/" rel="nofollow">http://status.squarespace.com/</a>
This shows great dedication on behalf of the team to provide a temporary solution to a more permanent problem. Well done.<p>More importantly, though - and not to discredit any of the hard work that's been done - hopefully the companies take a look at why the problem was created in the first place. For instance: why were the generators on the 17th floor? Why were the pumps below ground? Why was the datacenter built in a floodzone in the first place?<p>This is not unlike a lot of problems we face in software - developers bearing the consequences of poor planning.
I use Fastmail.fm for email and they're hosted at NYI, which seems to be fine. I wonder what their facilities are like?<p><a href="http://www.nyistatus.com/" rel="nofollow">http://www.nyistatus.com/</a>
Why are the backup generators on the 17th floor and not the 3rd floor? Assuming there is a very good reason for that,<p>Why wasn't there an additional pumping room on the 3rd floor, pre-built, with a legal amount of diesel in reserve, and a additional pumps to take over from the basement pumps when those fail, thus saving your bucket brigade 14 floors of climbing?<p>Why are you carrying diesel in the open in 5 gallon buckets and not in fuel containers that were purchased years ago?<p>All in all seems somewhat half-assed.
Can you use the elevators? If the generator doesn't have the extra power to run them, offer some customers credit and a mention in the post-mortem if they'll let you shut them down temporarily to power the elevators. Then you can bring fuel up in drums instead of buckets.<p>edit: Nope. <a href="http://news.ycombinator.com/item?id=4723814" rel="nofollow">http://news.ycombinator.com/item?id=4723814</a><p>:(
With all of this effort put into keeping the data center running, I've been wondering about a few things.<p>Was it actually connected to the outside world throughout the storm?<p>I have a hard time imaging that with the power out in large sections of the city some key router on the line wouldn't have also lost power.<p>If that's the case, the effort was put in just to keep the computers warm to prevent unplanned shutdown, not to actually provide uninterrupted service to the customer?<p>I'm not familiar with data center operation. If you're already cut off from the larger network at what point does it make sense to keep the machines running vs. shutting them down?<p>Or perhaps i'm just mistaken and they were actually connected throughout. In which case I find it amazing that the water knocked out pumps and necessitated other shutdowns but their network wasn't damaged in some way.
This story reminds me of the efforts one guy in New Orleans went through to keep his data center running after Katrina. There were some tales of diesel hauling in that blog as well.<p><a href="http://interdictor.livejournal.com/57475.html" rel="nofollow">http://interdictor.livejournal.com/57475.html</a>
<a href="http://interdictor.livejournal.com/40720.html" rel="nofollow">http://interdictor.livejournal.com/40720.html</a><p><a href="http://en.wikipedia.org/wiki/Interdictor_%28blog%29" rel="nofollow">http://en.wikipedia.org/wiki/Interdictor_%28blog%29</a>
Some numbers: a 1 megawatt generator burns 70 gallons / hour. If someone can carry 10 gallons (60 lbs), they need to make 7 trips / hour up 17 stories. I think one soldier could manage it.
The long-term correct solution to this problem is cloud infrastructure with multi-provider failover. If you have a server in California hosted by Amazon and a server in Texas hosted by Rackspace it's unlikely that you'll find yourself hauling diesel fuel up a staircase.
This got mentioned on an irc channel a few hours ago and my response was: "Why don't they build a pulley? They're nerds, they have the skills". Obviously shifting diesel about has some risks involved but a basic pulley system would help save a lot of time.
Any bets as to whether they will still be singing that common 'implementation efficiency doesn't matter, you can always scale horizontally' tune afterwards?