In a whistleblower statement to congress, Peter "Mudge" Zatko mentions that Twitter lacks plans and processes to "cold boot"<p>56. Cascading data center problems: In or around the spring of 2021, Twitter's primary data center began to experience problems from a runaway engineering process, requiring the company to move operations to other systems outside of this datacenter. But, the other systems could not handle these rapid changes and also began experiencing problems. Engineers flagged the catastrophic danger that all the data centers might go offline simultaneously. A couple months earlier in February, Mudge had flagged this precise risk to the Board because Twitter data centers were fragile, and Twitter lacked plans and processes to "cold boot." That meant that if all the centers went offline simultaneously, even briefly, Twitter was unsure if they could bring the service back up. Downtime estimates ranged from weeks of round-the-clock work, to permanent irreparable failure.
It the power generating world, this is called a Black Start condition, and plans exist to handle it. Given the nature of the "profit uber alles" atmosphere of the previous management, it's not really surprising that this is the case.<p>They should at least give everyone in the core engineering staff a burner phone and a printed (on paper, laminated) list of all the others, for use ONLY in the event of a blackout. Also perhaps, they should install a Starlink terminal for each data center, unconnected to all the other networking, with a stack of USB sticks for copying boot code?