"an initial verification led us to believe the changes had been rolled out successfully"<p>I would love more detail in the type II error in this validation step, and is worth exploring deeper. What was the verification step? Why did it not detect the issue? What review process was used for the verification step?<p>While the failed verification step is not the root cause, having good safety checks are the most important part of planning good changes, whether they're DNS reconfigurations, network changes or software deployments.