> Unbeknownst to us, the CDN Admin service was having a real bad time. It wasn’t prepared to handle the explosive level of traffic we were sending to them, and we were teetering on the edge of their own complete outage. [...]<p>> Without any end in sight, that team in charge of the Admin service did what they thought was their best option: they revoked our API token.<p>> It didn’t take long before the user complaints came filing in and the status page was updated alerting users to the situation. When the team realized what our situation had spiraled into, we immediately jumped into disaster recovery mode.<p>This sounds like a total communication failure to me! They are two teams of the same company, but they behave like complete strangers!<p>The API team started flooding CDN Admin endpoint, but didn't let them know -- couldn't they pop in into team's channel and let them know? And it sounds like the CDN Admin did not even try to communicate to API team before revoking their token? And even once the token was revoked, they did not let API team know ... instead API team had to look at user complains (why? wasn't there monitoring?)<p>This is pretty sad story which shows an importance of internal communications in a company.