<a href="https://status.cloud.google.com" rel="nofollow">https://status.cloud.google.com</a><p><a href="https://status.cloud.google.com/incident/compute/19003" rel="nofollow">https://status.cloud.google.com/incident/compute/19003</a><p>Status page reports all green, however the outage is affecting YouTube, Snapchat, and thousands of other users.
Disclosure: I work on Google Cloud (but disclaimer, I'm on vacation and so not much use to you!).<p>We're having what appears to be a serious networking outage. It's disrupting everything, including unfortunately the tooling we usually use to communicate across the company about outages.<p>There are backup plans, of course, but I wanted to at least come here to say: you're not crazy, nothing is lost (to those concerns downthread), but there is serious packet loss at the least. You'll have to wait for someone actually involved in the incident to say more.
Now is a good time to point out that the SLA of Google Cloud Storage only covers HTTP 500 errors: <a href="https://cloud.google.com/storage/sla" rel="nofollow">https://cloud.google.com/storage/sla</a>. So if the servers are not responding at all then it's not covered by the SLA. I've brought this to their attention and they basically responded that their network is never down.
There goes 3 nines for June and for Q2. I guess everyone gets a 10% discount for the month? <a href="https://cloud.google.com/compute/sla" rel="nofollow">https://cloud.google.com/compute/sla</a>
You know this reminds me of a bad taste that Google Sales team left when I asked for some of my billing that I was unaware of running after following a quickstart guide.<p>AWS refunded me in the first reply on the same day!<p>GCP sales rep just copy pasted a link to a self support survey that essentially told me, after a series of YES or NO questions that they can't refund me.<p>So why not just tell your customers like it is? Google Cloud is super strict when it comes to billing. I have called my bank to do a chargeback and put a hold on all future billing with GCP.<p>I'm now back to AWS and still on a Free Tier. Apparently the $300 Trial with Google Cloud did not include some critical products, AWS Free tier makes it super clear and even still I sometimes leave something running on and discover it in my invoice....<p>I've yet to receive a reply from Google and its been a week now.<p>I do appreciate other products such as Firebase but honestly for infrastructure and for future integration with enterprise customers I feel AWS is more appropriate and mature.
GCP status page is worthless as it's always happy and green when production systems are down and then they might acknowledge something an hour later
Google Cloud is the number 4 most monitored status page on StatusGator and Google Apps is number 12. In addition, at least 20 other services we monitor seemingly depend on Google Cloud because they all posted issues as soon as Google went down.<p>It's always interesting to see these outages at large cloud providers spider out across the rest of the internet, a lot of the world depends on Google to stay up.
And thus was ruined hundreds or thousands of pleasant Sunday afternoons.<p>I don’t miss being on pager duty one bit. I see it looming in my headlights, sadly.
That feeling when you open <a href="https://console.cloud.google.com" rel="nofollow">https://console.cloud.google.com</a> and see that you don't have your Kubernetes clusters and CloudSQL databases, but CTA to create first.
Nest is down too, not surprising given they are part of Google. What I don't understand is why I can't still control my devices over my local network. Why does the system even require access to Google servers?
It seems the AdWords anti-spam system is down, which means anyone can put a billion dollar bid on every keyword and get their ads showing on every Google search for every query.<p>Systems that fail 'open'...
Level(3) one of the biggest backbones has issues too,
might be related<p><a href="https://downdetector.com/status/level3" rel="nofollow">https://downdetector.com/status/level3</a>
Funny how as soon as I realized that Gmail and Google Sheet aren’t working properly I rushed to HN to figure out what’s going on. I love this community!
The two Google Cloud networking incidents are:<p>Incident #19008 began at 2019-06-02 12:48. <a href="https://status.cloud.google.com/incident/cloud-networking/19008" rel="nofollow">https://status.cloud.google.com/incident/cloud-networking/19...</a><p>Incident #19009 began at 2019-06-02 12:53. <a href="https://status.cloud.google.com/incident/cloud-networking/19009" rel="nofollow">https://status.cloud.google.com/incident/cloud-networking/19...</a><p>Times are US/Pacific
I also noticed that Google search stop indexing news articles.<p>So I searched for "gmail down" on bing and I got some results [1]. But searching on Google for "gmail down" does not return any results [2].<p>[1] <a href="https://www.bing.com/news/search?q=gmail+down&qs=n&form=QBNT" rel="nofollow">https://www.bing.com/news/search?q=gmail+down&qs=n&form=QBNT</a><p>[2] <a href="https://www.google.com/search?q=gmail+down&source=lnms&tbm=nws" rel="nofollow">https://www.google.com/search?q=gmail+down&source=lnms&tbm=n...</a>
And Gmail too doesn't feel very well today.<p><pre><code> [21:55:19] POP< +OK send PASS
[21:55:19] POP> PASS ********
[21:55:21] POP< +OK Welcome.
[21:55:21] POP> STAT
[21:55:21] POP< -ERR [SYS/TEMP] Temporary system problem.
Please try again later.</code></pre>
I was playing around this afternoon with appengine, and thought I broke one my projects when I started getting 502 back.<p>There appears to be some irregularities on consumer services as well that are of course certainly related, youtube was behaving a bit oddly for me.<p>The impact seems to be cascading down from just GCE to other services as well - that status page certainly does not reflect the reality of the situation. You can't even sign into GCP right now, and things that run on GCE, like appengine seem impacted.
Yep, I can no longer see my Cloud SQL database - it's as if I've never created one at all. Really hoping this is just an issue displaying it and that Google hasn't punted my infrastructure and backups.
Other google services are also affected:<p><a href="https://www.google.com/appsstatus#hl=en&v=status" rel="nofollow">https://www.google.com/appsstatus#hl=en&v=status</a><p><a href="https://i.imgur.com/pcqwwA4.png" rel="nofollow">https://i.imgur.com/pcqwwA4.png</a>
It seems crazy to me that Google Cloud can have this level of instability but I, on the other hand, can never remember google.com going down.<p>Why are they operating one with a different networking infrastructure from the other?
Right now this is at the 'good at least it's not mine to fix or worry about' kind of like 'and the reason I choose IBM' [1]. I can just sit back and wait for gmail to work correctly. Now at the point it starts to last what I would consider a long time well then I will have things to worry about.<p>One thing with gmail though. When it's down it's similar to a snow storm if you only do business in a city. Everyone is impacted and everyone understands a missed deadline is unavoidable.<p>[1] For those not old enough to know what I mean read this: <a href="https://www.ibm.com/ibm/history/ibm100/us/en/icons/personalcomputer/words/" rel="nofollow">https://www.ibm.com/ibm/history/ibm100/us/en/icons/personalc...</a>
Looks like my Gmail is back, but I don't have any emails from while it was down. Yikes.<p>Edit: just got one email from the downtime, so perhaps my initial conclusion was incorrect
We're hosting an open global Zoom call for all engineers affected by the outage, join us at <a href="https://zoom.us/j/793450725" rel="nofollow">https://zoom.us/j/793450725</a>
Does anyone know if this is a regional or global outage?<p>I can see my GKE clusters in one region but not in another, so I am guessing it's the former.<p>Looks like we'll need a cluster in each region going forward...
I tried multiple times to setup a Google Wifi router today. Wifi would work but the app said it was offline. Perhaps I am not insane or incompetent after all
<a href="https://status.cloud.google.com/incident/compute/19003" rel="nofollow">https://status.cloud.google.com/incident/compute/19003</a> Does status update mean only the status. Won't there be words like "sorry", "apologize" & "inconvenience"! Only PR is responsible for those words?
Not sure if related, but I was going to a BBQ yesterday and myself and 3 other people got lost because Google Maps app glitched out, directing us to the wrong places. If you search twitter for #googlemaps tons of people have the same issue. Surprised no one has posted about it.
<a href="https://status.cloud.google.com/incident/compute/19003" rel="nofollow">https://status.cloud.google.com/incident/compute/19003</a><p>>We will provide an update at 16:00 US/Pacific.<p>it's 16:22 and no updates were posted. a bit unprofessional..
I appears that logging into the webmail solves at least POP mail problems. I tried my mail client and failed, then attempted a login to the webmail which worked. Gmail then asked me to confirm my recovery address and cellphone, which I did, and finally loaded the inbox page. I immediately attempted a connection through the POP client and this time it worked.<p>It might be something security related if it triggers a mandatory identity confirmation.<p>edit: I tried to send me a mail from another account and it worked but out of 4 or 5 mail checks at least two failed giving the same error.<p>[23:44:27] POP< -ERR [SYS/TEMP] Temporary system problem. Please try again later.<p>The problem seems much more complex.
That explains why my google home thing thinks it’s sub-zero even though I’m warm in shorts and a t-shirt.<p><a href="https://pasteboard.co/IhBsyrsO.jpg" rel="nofollow">https://pasteboard.co/IhBsyrsO.jpg</a>
The last big outage, iirc, was Google didn't test their rollback procedures for router upgrades. I'll be very interested to hear if it's yet another change control problem that caused this outage.
I was finishing a university assignment with the deadline 90mins away.<p>I wanted to upload a video of the project to YouTube and add a link to it in the report. YouTube takes a long time to process the video, and then says it's unavailable.<p>I go to Vimeo: it's down.<p>I upload the video to Dropbox, and copy its link to the report.<p>But my report was a Google doc. And when I tried to export it as PDF (which I had not done yet) it couldn't do it. I never hated google more.<p>Eventually the video went through to YouTube, and I could export the PDF on the third try, but this really made me conscious of my dependance on Google.
Confirming issues on our end. I'm able to load up my console but when I go to Kubernetes Engine, I don't see my clusters. I'm monitoring closely on twitter
Was certainly an interesting alert when my Cloud Functions started reporting downtime. Among the many things that dip in and out on what seems like a monthly basis, I’ve not seen them just drop out in quite a while. Hopefully they get things sorted out. I can’t really imagine what it looks like internally when this level of outage is going on, but I want to think everyone is fairly collected
My builds are failing because it cannot download the chromium.<p>> Error: Download failed: server returned code 502. URL: <a href="https://storage.googleapis.com/chromium-browser-snapshots/Linux_x64/662092/chrome-linux.zip" rel="nofollow">https://storage.googleapis.com/chromium-browser-snapshots/Li...</a>
Does Google also have some sort of listing on which consumer apps are particularly affected (i.e. Gmail, Hangouts, Docs, Sheets, etc).<p>The cloud components may be directly affected but for consumers, there's nothing which will provide info on what consumer facing services are getting some issues.
Majority of G Suite services are suffering service outages: <a href="https://www.google.com/appsstatus#hl=en&v=status" rel="nofollow">https://www.google.com/appsstatus#hl=en&v=status</a>
Anyone experiencing issues with GCS? Seems highly intermittent and dependent on the location the request is coming from (maybe that's because it's a networking bug).<p>The status page says GCS is fine but that's highly unlikely.
Took me a while to track latency issues to GCP. Wasn't expecting it. This also seems to affect some GAE instances and some of their products like google photos. At least according to my observations
I just can't reach google apps on my HTC m9 since yesterday. I am in West Africa. My Whatsapp crashed too and I lost all my previous threads.
Is my issue related to Cloud being down?
Playing "is google on this website or not?" just became so much easier, simply see if website works or not.<p>Scary stuff. What happens when Murphy's law decides to crash things even more?
Yeah I was having trouble accessing my Gsuite apps, had a couple of 502s, which led me to check HN. While it doesn't give me 502 now, it's abnormally slow.
Was trying to set up SSL on a GKE cluster today. Guess I'll have to wait for tomorrow if I want to be able to tell my mistakes apart from Google's.
u.s. west: all our cloud compute is inaccessible rn.... our API is down, can't ssh into the servers, and also can't see them on the dashboard.
With Google Cloud incidents, most of the time whole regions fail, and with AWS generally only a region fails. Of course there would be exceptions, but Google Cloud does not make me feel safe as an outsider (and a user of multi-region AWS)
These things happen. That's OK. Here's what's not OK:<p>>
We are investigating an issue with Google Compute Engine. We will provide more information by Sunday, 2019-06-02 12:45 US/Pacific.<p>The next update is at 12:59. Just ... no.
Prediction: the final postmortem will say "someone pushed a bad config", just like most of the previous postmortems (and most of the internal postmortems as well, for Borg-based services). This is the cause of most other outages in other cloud providers as well. A really hard to solve problem.
What I've realized from this: Google doesn't have an official status page for GCP. There are a few unofficial ones, but nothing official that I could find.