AWS us-east-2 outage

Sorry all I jinxed it. Yesterday I was in a meeting and said &quot;The only regional outages AWS has ever had were in us-east-1, so we should just move to us-east-2.&quot;<p>Now I guess we have to move to us-west-2. :)<p>Update: looks like it&#x27;s only one zone anyway, so my statement still stands!

The fact that so many popular sites&#x2F;services are experiencing issues due to a single AZ failure makes me think that there is a serious shortage of good cloud architects&#x2F;engineers in the industry. It would be one thing if this was a Regional failure, but a single AZ failure should not have any noticeable effect.

Update from AWS: they lost power to (part of?) a single DC in the use2-az1 availability zone.<p>10:25 AM PDT We can confirm that some instances within a single Availability Zone (USE2-AZ1) in the US-EAST-2 Region have experienced a loss of power. The loss of power is affecting part of a single data center within the affected Availability Zone. Power has been restored to the affected facility and at this stage the majority of the affected EC2 instances have recovered. We expect to recover the vast majority of EC2 instances within the next hour. For customers that need immediate recovery, we recommend failing away from the affected Availability Zone as other Availability Zones are not affected by this issue.

And I thought us-east-2 is the way to escape us-east-1&#x27;s problems.

<p><pre><code> $ dig news.ycombinator.com ;; ANSWER SECTION: news.ycombinator.com. 1 IN A 50.112.136.166 $ dig -x 50.112.136.166 ;; ANSWER SECTION: 166.136.112.50.in-addr.arpa. 300 IN PTR ec2-50-112-136-166.us-west-2.compute.amazonaws.com. </code></pre> saving couple keypresses just in case

I find this spreadsheet handy for thinking about AWS region-wide outages and frequency. This seems to be the first major us-east-2 outage, indeed, vs us-east-1 and other regions.<p><a href="https:&#x2F;&#x2F;docs.google.com&#x2F;spreadsheets&#x2F;d&#x2F;1Gcq_h760CgINKjuwj7WuRmLXHIdvsUdzNQCg0g4QvVs&#x2F;edit" rel="nofollow">https:&#x2F;&#x2F;docs.google.com&#x2F;spreadsheets&#x2F;d&#x2F;1Gcq_h760CgINKjuwj7Wu...</a> (from <a href="https:&#x2F;&#x2F;awsmaniac.com&#x2F;aws-outages&#x2F;" rel="nofollow">https:&#x2F;&#x2F;awsmaniac.com&#x2F;aws-outages&#x2F;</a>)

On the bright side, this is certainly a good test for &quot;exactly how resilient are our AWS-based systems to the loss of a single availability zone?&quot;

Ahh<p>Always check HN before trying to diagnose weird issues that shouldn&#x27;t be connected

Just lost my email provider (<a href="https:&#x2F;&#x2F;status.postmarkapp.com&#x2F;incidents&#x2F;240161" rel="nofollow">https:&#x2F;&#x2F;status.postmarkapp.com&#x2F;incidents&#x2F;240161</a>) to this and I&#x27;d bet my services are degraded&#x2F;down. I know it&#x27;s never a &quot;good&quot; time for an outage but this sure does suck for me right now. We&#x27;ve got an event this weekend and people can&#x27;t sign up right now to buy tickets&#x2F;etc.

AWS reported the outage here : <a href="https:&#x2F;&#x2F;health.aws.amazon.com&#x2F;health&#x2F;status" rel="nofollow">https:&#x2F;&#x2F;health.aws.amazon.com&#x2F;health&#x2F;status</a>

I dunno how else to put it. Having EVERYTHING on AWS is a national security threat.<p>This isn&#x27;t good, and someone who can do something about it needs to.

Suppose it&#x27;s time to setup multi-az and pay to insure against AWS&#x27; own failures. I don&#x27;t know why I previously thought their EC2 uptime claims were sufficient. Lesson learned.

AWS acknowledged the issue <a href="https:&#x2F;&#x2F;health.aws.amazon.com&#x2F;health&#x2F;status" rel="nofollow">https:&#x2F;&#x2F;health.aws.amazon.com&#x2F;health&#x2F;status</a>

Thanks for sharing. We&#x27;ve just spent the last hour debugging our website, thinking we had issues. This explains it.

We have always been at war with us-east-2.

I&#x27;m running Terraform and it appears to be stuck now. What do I do??

us-east-2 customer here, having a variety of &quot;strange&quot; issues including inability to reach an RDS database, other users in my firm having VPN reconnect trouble to that region.

Zoom is having connectivity issues. Nothing on <a href="https:&#x2F;&#x2F;status.zoom.us&#x2F;" rel="nofollow">https:&#x2F;&#x2F;status.zoom.us&#x2F;</a> page yet.

Anyone&#x27;s ECR endpoints went out during the outage? We&#x27;ve had timeouts while pulling images onto our k8s cluster post-restart

Wonder if this is why Zoom is down. Wasn&#x27;t able to connect just now. The connection proxy&#x2F;sites were giving 504s.

I just set up a few small sites (not live yet) on us-east-2, because us-east-1 has a poor reputation. I wanted to avoid multi-region to keep things simple, but now I&#x27;m thinking I might have to spend the additional time on it. Not ideal when there&#x27;s no dedicated ops.

Same. Our EC2 instances can&#x27;t connect RDS and just got 500 errors on the dashboard.

Can confirm based on what Metrist is seeing. Looks to be a larger issue in the US East, also seeing Cloudfare and Datadog with issues.

I&#x27;ve somehow dodged region outages on AWS for years, and here&#x27;s my first one. So many alerts firing off in unexpected ways.

Same here - we were finally able to log in to the console, but we&#x27;re in us-east-2 and are having a ton of issues.

Tons of issues in us-east-2 here as well

Personally only saw intermittent failures through out. Rather minor production as far as AWS outages go.

No wonder my error logs were clean. I can get into my hosts, but my LB isn&#x27;t routing. Sad face.

Lots of issues in us-east-2 for instances for us but also other regions when connecting to RDS

Lots of things intermittently unreachable in us-east-2 for us, across multiple AWS accounts.

Chaos monkey is on the move again.

Anyone else notice similar issues in US-West-2 a few hours before this issue in US-East-2?

Looks like a lot of services are impacted including Cloudflare, Ping, Zoom and Datadog.

unfortunately one of the largest regions:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;patmyron&#x2F;cloud&#x2F;#ip-addresses-per-region" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;patmyron&#x2F;cloud&#x2F;#ip-addresses-per-region</a>

I&#x27;m seeing issues in 2a but not 2b. Anyone having issues in 2b?

Had Rackspace login issues earlier today. Hmmmmmm ...

us-east-2 customer here also having some issues

We have RDS and ECS issues in us-east-2

yeah looks like us-east-2 has networking issues

Can confirm....

Same

I understand that us-east is AWS&#x27;s oldest and biggest facility, but Amazon seems to have more money than Croesus, why aren&#x27;t they fixing&#x2F;rebuilding&#x2F;replacing us-east with something more modern?