> As to what can be done to prevent similar failures, the FCC is recommending CenturyLink and other backbone providers take some basic steps, such as disabling unused features on network equipment, installing and maintaining alarms that warn admins when memory or processor use is reaching its peak, and having backup procedures in the event networking gear becomes unreachable.<p>Disabling unused services? Alarms when nearing resource limits? Contingency plans? How is this the first time this has come up?! These are like security & devops 101.
Network engineer here: clink bridged all of the management controllers on their infinera dwdm shelves together into one multi state sized L2 broadcast domain. Best guess is because it made them easier to SNMP poll and to run other management tools to admin them.<p>Within the circle of people who really know what went on, we've been laughing at them for months.
Page 7 -<p>>><i>CenturyLink and Infinera state that, despite an internal investigation, they do not know how or why the malformed packets were generated.</i><p>So we still don’t know why the rotten packets were created in the first place?
Correct title is: how misconfigured century link network broke when rotten packet arrived.<p>This title sounds like it was packet failure, while it is not, it was a matter of time until this problem occurs, hardware must be resilient to malformed input.