TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Google Cloud networking issues in us-east1

586 点作者 decohen将近 6 年前

31 条评论

boulos将近 6 年前
Disclosure: I work on Google Cloud (but I&#x27;m not in SRE, oncall, etc.).<p>As the updates to [1] say, we&#x27;re working to resolve a networking issue. The Region isn&#x27;t (and wasn&#x27;t) &quot;down&quot;, but obviously network latency spiking up for external connectivity is bad.<p>We are currently experiencing an issue with a subset of the fiber paths that supply the region. We&#x27;re working on getting that restored. In the meantime, we&#x27;ve removed almost all Google.com traffic out of the Region to prefer GCP customers. That&#x27;s why the latency increase is subsiding, as we&#x27;re freeing up the fiber paths by shedding our traffic.<p>Edit: (since it came up) that also means that if you’re using GCLB and have other healthy Regions, it will rebalance to avoid this congestion&#x2F;slowdown automatically. That seemed the better trade off given the reduced network capacity during this outage.<p>[1] <a href="https:&#x2F;&#x2F;status.cloud.google.com&#x2F;incident&#x2F;cloud-networking&#x2F;19016" rel="nofollow">https:&#x2F;&#x2F;status.cloud.google.com&#x2F;incident&#x2F;cloud-networking&#x2F;19...</a>
评论 #20342285 未加载
评论 #20339796 未加载
评论 #20339866 未加载
评论 #20339559 未加载
评论 #20340747 未加载
评论 #20340518 未加载
harshreality将近 6 年前
Hacker News: The real status page and help desk for the internet.<p>Do companies realize how absurd this is?<p>ETA: It seems someone at Google had a change of heart, and most of what boulos posted in this thread has been added as updates to the official google status page. Better late than never, I guess, especially if this is the start of a trend in outage reporting.
评论 #20339337 未加载
评论 #20342500 未加载
评论 #20339298 未加载
评论 #20339382 未加载
评论 #20341214 未加载
fastest963将近 6 年前
Here&#x27;s the original issue: <a href="https:&#x2F;&#x2F;status.cloud.google.com&#x2F;incident&#x2F;cloud-networking&#x2F;19015" rel="nofollow">https:&#x2F;&#x2F;status.cloud.google.com&#x2F;incident&#x2F;cloud-networking&#x2F;19...</a><p>Not sure why they closed that one at 9:12 just to open a new one at 10:25. We didn&#x27;t see any traffic coming to us-east1 during that time period so I would assume the original issue is still the root cause.
评论 #20339239 未加载
评论 #20338431 未加载
mehrdadn将近 6 年前
Does anybody else feel like there have been a lot of outages in recent months? And I don&#x27;t mean Google -- I mean lots of others too (I seem to recall CloudFlare, Facebook, etc.)... are they really increasing or are we just hearing more about them? Seems a bit odd.
评论 #20340321 未加载
评论 #20340204 未加载
评论 #20340449 未加载
评论 #20342250 未加载
评论 #20340057 未加载
评论 #20340848 未加载
评论 #20341633 未加载
评论 #20340102 未加载
Thaxll将近 6 年前
Looks like an external issue. &quot;The Cloud Networking service (Standard Tier) has lost multiple independent fiber links within us-east1 zone. Vendor has been notified and are currently investigating the issue.&quot;
评论 #20338630 未加载
评论 #20338595 未加载
thsowers将近 6 年前
Why so many problems at Google lately? Calendar down two weeks ago[0], and Google Cloud had a larger outage a month ago[1]<p>[0]: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=20213092" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=20213092</a><p>[1]: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=20077421" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=20077421</a>
评论 #20338875 未加载
评论 #20338380 未加载
评论 #20339631 未加载
评论 #20338802 未加载
评论 #20338554 未加载
mrmattyboy将近 6 年前
To whomever commented something like &#x27;laughs in AWS&#x27; (comment was removed before I submitted the comment)...<p>please don&#x27;t...<p>glass house and all that... but I also share the same glass house as you.. I don&#x27;t want bad luck<p>... and it&#x27;s only a fluke that this happened to google in eu-east1 and not AWS in X region and then you (and I) would be having a time of hell! :&#x2F;
评论 #20338691 未加载
评论 #20338606 未加载
评论 #20338957 未加载
inlined将近 6 年前
Holy crap. It’s an outage in all zones? What’s the point of AZs if you lose whole DCs at a time.
评论 #20339843 未加载
评论 #20338418 未加载
评论 #20338747 未加载
评论 #20338656 未加载
username444将近 6 年前
Cloudflare was returning a 502 this morning, wonder if they&#x27;re related. Lots and lots of sites down for about an hour, including all of Shopify.
评论 #20339295 未加载
评论 #20338768 未加载
评论 #20338809 未加载
评论 #20339521 未加载
评论 #20338849 未加载
estsauver将近 6 年前
In a moment that&#x27;s likely to be very, very frustrating for a large number of you that have businesses and customers that depend on G cloud, let&#x27;s try to remember that somewhere there&#x27;s an engineer or an SRE having a really hard day just trying to fix things.<p>Please, be kind and decent to each other, especially when things are hard.
评论 #20338498 未加载
评论 #20338540 未加载
评论 #20338468 未加载
评论 #20338495 未加载
pupdogg将近 6 年前
&gt; The disruptions with Google Cloud Networking and Load Balancing have been root caused to physical damage to multiple concurrent fiber bundles serving network paths in us-east1.<p>I am assuming some sort of construction zone at or nearby the facility and the backhoe operator dug in and accidently cut the cables?
verdverm将近 6 年前
I&#x27;ve been working out of us-east1 all day and haven&#x27;t noticed
partiallypro将近 6 年前
It&#x27;s been down for 4 hours and it&#x27;s just now being posted on HN? Is it intermittent?
评论 #20339359 未加载
评论 #20338664 未加载
digitalsanctum将近 6 年前
I routinely see notices of outages like this posted on HN while HN itself never seems to be impacted. This begs the question: Where and how is HN hosted in a way that avoids being impacted by widespread network and provider outages?
评论 #20340361 未加载
评论 #20340411 未加载
pkaye将近 6 年前
Looks like all that high end engineering talent and processes still has its limits.
dragonwriter将近 6 年前
&gt; The disruptions with Google Cloud Networking and Load Balancing have been root caused to physical damage to multiple concurrent fiber bundles<p>Is this concurrent damage to separated bundles or damage to colocated bundles?
saltminer将近 6 年前
The title says &quot;almost 4 hours&quot; (was posted at around 3 PM EST), but the incident was created at 10:25 AM PST, which is 1:25 PM EST. Has it been more like 2 hours or is there more to this incident?
bob33212将近 6 年前
Down or just high latency? For some folks that is the same thing.
评论 #20338683 未加载
hnaccy将近 6 年前
What&#x27;s the actual number of 9s for the major cloud services these days?<p>My impression from their PR seems to mismatch the number of outages and issues lately.
评论 #20338610 未加载
评论 #20338627 未加载
评论 #20338567 未加载
评论 #20338736 未加载
评论 #20338633 未加载
wwwpppddd将近 6 年前
App Engine and Cloud functions were apparently returning error rates of &gt; 30 percent overall between 11 a.m. and 3 p.m., with some projects experiencing a 100 percent error rate. GCS was also experiencing issues for the first half, which was attributed to the networking issues. Google said the networking issues were resolved initially but then stated they were investigating the GAE issues. Those issues were resolved, and the networking issue has been reopened as of 2:35 eastern: <a href="https:&#x2F;&#x2F;status.cloud.google.com&#x2F;incident&#x2F;cloud-networking&#x2F;19016" rel="nofollow">https:&#x2F;&#x2F;status.cloud.google.com&#x2F;incident&#x2F;cloud-networking&#x2F;19...</a>.<p>GAE and all other services still show green here, of course: <a href="https:&#x2F;&#x2F;status.cloud.google.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;status.cloud.google.com&#x2F;</a>
noncoml将近 6 年前
Bad config push again?
评论 #20338880 未加载
评论 #20338559 未加载
评论 #20338556 未加载
z3t4将近 6 年前
When choosing a big cloud provider people forget that it&#x27;s many orders of magnitude more complicated to run something at Google scale then to maintain <i>one</i> single server. For example the whole Stack overflow website runs on one or two servers. World of Warcraft also used to run on one single (blade) server. Chances are one server will be good enough for most use cases. And if you don&#x27;t want to have it in your closet there are plenty of dedicated hosting and colocations.
评论 #20340249 未加载
评论 #20339945 未加载
评论 #20339649 未加载
mountainofdeath将近 6 年前
Another day, another Google outage. It feels like it&#x27;s once a month this year
rco8786将近 6 年前
2019 has been a really rough year for GCP
geogram将近 6 年前
Longer than 4 hours. We have stackdriver setup to monitor uptime&#x2F;latency and its been acting up since 2am PST.
评论 #20339219 未加载
garyb2将近 6 年前
Notice they did not get around posting the next status update on time.
awinter-py将近 6 年前
do they not have extra hands on staff to dedup the messages? what&#x27;s with the identical messages at 14:31, :44, :48? This happened last time too.
评论 #20340706 未加载
lgats将近 6 年前
Pretty sure I&#x27;ve read before that us-east1 is one of the older Google data centers presumably with older equipment
评论 #20339351 未加载
spullara将近 6 年前
Wow. GCP is always a networking issue. Their QA on networking changes needs work. Maybe they should spend 20% on it.
dx87将近 6 年前
Kind of related to this, but these types of outages are why I moved from Google Play to Spotify for streaming music. Their infrastructure seems so large that things that should be a standalone service, like streaming music, are bound to be collateral damage when they mess something up on another service. Having everything provided by one company is convenient until it all goes down at the same time and you can&#x27;t access your email, videos, or music because they all run on the same infrastructure.
评论 #20338827 未加载
评论 #20339089 未加载
评论 #20338851 未加载
评论 #20341071 未加载
codingslave将近 6 年前
Google engineers ran into a coding problem that wasnt on leetcode
评论 #20341914 未加载
评论 #20339164 未加载