Google Cloud Networking Incident

We&#x27;ve been having &#x27;fun&#x27; with ongoing issues for a site since 6pm UTC yesterday, which got dramatically worse this morning...and having been recurring during the day.<p>Having multiple hour outages makes me really want to go back to hiring a couple of physical servers in a rack somewhere.

For anyone wanting a quick workaround, try removing all the backend of your load balancer then add them again. No idea why but it did work for us.

Just to verify my understanding:<p>* This is a multi-region (us-east1, us-central1, europe-west1, asia-northeast1 and asia-east1) outage (yeesh)<p>* Incident began at Incident began at 2017-08-29 18:35 PDT. At the time of this comment, we&#x27;re over 13 hours (??)<p>Correct?

&quot;A distributed system is one in which the failure of a computer you didn&#x27;t even know existed can render your own computer unusable.&quot; — Leslie Lamport<p>I think Leslie needs to update this for Cloud, however.

Our load is not being &quot;balanced&quot;, as not all backends are being utilized because of this.

we&#x27;ve been suffering from this since 3:07am UK time and it&#x27;s sad to see google still hasn&#x27;t learnt how to do support&#x2F;communicate with their paying customers after all the flack they always take about this.<p>The number of mistakes (silly and otherwise) keeps increasing and I&#x27;m not talking about the technical problem itself! ok, end of rant

Stuff goes wrong everywhere. The only problem is thinking that any cloud vendor or service is 100% reliable.

Does this affect the global load balancer too?<p>We haven&#x27;t seen any issues, so presumably not? Or is it just a subset?

Whenever the sun starts shining, the clouds disappear.

Yikes! that&#x27;s a huge failure.