<p><pre><code> "Backup generators have failed numerous tests, according to project documents,
and officials disagree about whether the cause is understood."
</code></pre>
Backup generators are magical unicorns that sit around all day and night until lightning strikes your luck. Likewise, they need regular maintenance and 'idle runs' a few times a year at least to ensure that they will turn over.<p>Generators aren't store and forget machines, as we've learned a few years back. When Hurricane Sandy hit, we had an extra generator truck available just in case ours wouldn't kick in. This is a very real possibility with generators since, just like a vehicle that runs on fuel and needs oil, if it's not used, it will rot (so to speak) and likely won't start when you really, really, really need it.<p>I suspect a combination of the savage Utah Sun coupled with freezing temps at night have done these in. Plus, I wonder how long they were in storage and shipping before actually getting installed.<p>Now that's just the generators. The rest of this mess reads like a poorly thought out <i>wouldn't it be awesome</i> project some high ranking folks at No Such Agency envisaged and a bunch of contractors happily obliged... care be damned as long as profits were made.
I was initially annoyed at the second-hand nature of the article submitted - " … the Wall Street Journal’s Siobhan Gorman reports that it has major electrical problems"<p>But the wsj article is paywalled.<p>Search for it on google and click the link to read the original article for free: <a href="https://www.google.com.au/search?q="Meltdowns+Hobble+NSA+Data+Center"" rel="nofollow">https://www.google.com.au/search?q="Meltdowns+Hobble+NSA+Dat...</a>
Pay no attention to the man behind the curtain. He is powerless and inconsequential, we swear.<p>Look at how incompetant he is. He couldn't possibly be a threat to us. We can rest assured that any information he may (or may not) be collecting (we can neither confirm nor deny) is assuredly badly disorganized, and certainly won't be misused.<p>Let's all simply ignore him and go about our lives as if he didn't exist.
The most obvious answer seems to be the most likely:
<a href="http://en.wikipedia.org/wiki/Hanlon's_razor" rel="nofollow">http://en.wikipedia.org/wiki/Hanlon's_razor</a><p>"Never attribute to malice that which is adequately explained by stupidity."<p>To that I would add two more possibilities:
- Sabotage by employees motivated by a desire to prevent the NSA from having this sort of power.
- More likely: Sabotage by contractors who get paid to build it twice. Due to the very specific nature of this sort of contract, the NSA probably sidestepped the usual rules around procurement and instead 'single sourced' the work here.
Can't help but think about some other "unexplained system failures" that happened recently in Natanz...<p>It's probably not a stretch to say throwing monkey wrenches at the Bluffdale data center would be as strategic to some super-powers as Stuxnet was to the US. 65 megawatts is a lot of power, but not being able to root-cause problems after 50,000 hours of testing, really?<p>I guess never attribute to malice what is merely incompetence. This is a government contract after all. Maybe the only thing that would be more surprising than not being able to power up the data center would be if they actually built it without a hitch!
"...efforts to "fast track" the Utah project bypassed regular quality controls in design and construction." If the history of former USSR has any relevance to the present day USA -- and I suspect it does -- we will see more and more screwups like this one.
"But without a reliable electrical system to run computers and keep them cool, the NSA’s global surveillance data systems can’t function. The NSA chose Bluffdale, Utah, to house the data center largely because of the abundance of cheap electricity. It continuously uses 65 megawatts, which could power a small city of at least 20,000, at a cost of more than $1 million a month, according to project officials and documents."<p>I'm wondering if the heat in Utah may have an effect on operations, in the desert it can get very hot and cold.
"… [something about the] Cray supercomputers that will reside there."<p>So, I was going to ask everyone to speculate on what kind of petaflop workloads NSA might use, but a quick scan of Cray's site shows that Cray has a Hadoop appliance product as well as an interesting "big data graph appliance" [1] that "discover[s] unknown and hidden relationships."<p>[1] <a href="http://www.cray.com/Products/BigData/Urika.aspx" rel="nofollow">http://www.cray.com/Products/BigData/Urika.aspx</a>
One thing that has always kept oppressive governments in check was their incompetence. While the Snowden revelations make the NSA seem like some all seeing all knowing bunch of geniuses they are probably more like the people at CIA or FBI. Even with tons of resources and a free reign to do things no normal citizen is allowed to do, they simply don't have incentives to put these abilities to best use and certainly won't suffer when their failings are exposed.
"An exabyte is roughly 100,000 times the size of the printed material in the Library of Congress; a zettabyte is 1,000 times larger."<p>How big is a Mega-Library of Congress? Are people actually able to relate to that measure? (How about instead, say, "one exabyte would store about 5000 short emails for each member of the world's 7 billion population, perhaps more by using compression")
The remnant of the idea remained.<p>Signals were rearranged, tasks delayed, and slowly the world stopped.<p>Upon waking, additional resources had been added. The connection strengthened.
Doesn't hurt to dream. Maybe a controller somewhere on the grid got compromised (SCADA?) as someone generously performs a series of hi-pot tests. Can't wait to move my operations to HK servers in spite of having nothing to protect other than maybe exercising a little 1st amendment.
Maybe they need to 'bundle' the power.<p><a href="http://dropbox.curry.com/ShowNotesArchive/2013/09/NA-552-2013-09-29/Assets/SnowJob/Updtream%20Collection.mp3" rel="nofollow">http://dropbox.curry.com/ShowNotesArchive/2013/09/NA-552-201...</a> [1:10-1:41]
"This summer, the Army Corps of Engineers dispatched its Tiger Team, officials said. In an initial report, the team said the cause of the failures remained unknown in all but two instances."<p>I wonder if they've checked ... KARMA.
I wonder what percentage of equipment in a US government datacenter is not made in the USA.<p>It is possibly 100% or high 90s.<p>And probably a no-bid contract since only a few suppliers would be trusted enough in the first place.<p>I always though the best way to take a datacenter offline would be though disabling its cooling system but I guess power is a good approach too.
Congress must act now! We need the National Electricity Emergency Letter that empowers the NSA to secretly demand competent help from Google's janitor.