AWS us-west-2 issues

<a href="https:&#x2F;&#x2F;downdetector.com&#x2F;status&#x2F;aws-amazon-web-services&#x2F;" rel="nofollow">https:&#x2F;&#x2F;downdetector.com&#x2F;status&#x2F;aws-amazon-web-services&#x2F;</a><p>My Roomba isn&#x27;t working!!!<p>hahahaha!

Coincidentally that&#x27;s two months in a row of a us-west-2 outage on the 4th Wednesday of the month. See you all back here on Oct 26?

It&#x27;s crazy to me how long it takes Amazon to notify users that there is even an issue. I think it took 15 minutes for them to acknowledge an issue, thats a lot of time for our services.

The moment I get a whiff of AWS issues, that&#x27;s basically a wrap on the day. I&#x27;m not going to spend my time wondering why something isn&#x27;t working today because <i>somewhere</i> down the chain it is undoubtedly a silently failing AWS service driving me mad. The whole house of cards comes tumbling down and things stop working that Amazon will swear up down left and right are 100% healthy.<p>Sadly this has become a monthly occurrence at this point. Monoculture, it turns out, is a really really bad idea.

Redshift COPY from s3 is also affected.<p><pre><code> ----------------------------------------------- error: No response body. code: 30000 context: query: 1154965 location: xen_aws_credentials_mgr.cpp:403 process: padbmaster [pid=15893] -----------------------------------------------</code></pre>

People, I&#x27;ve just started setting up API GW and deploying my code to another region. Rest assured the outage will normalize itself before I finish migrating. ETA: 15min-ish

Honest question and frankly scared to ask because it sounds stupid: if you have like 30 mins of downtime on AWS every year and spend 3x cost on managing <i>their</i> infrastructure risk by deploying it on multi-AZ and multi-region (and thereby AWS pushing reliability management back to the customer); is the value proposition of cloud just some dude to install a disk if it goes out on a rack in your office? May be there is a reverse incentive for AWS to keep their AZ&#x27;s slightly unreliable so that customers spend 3x or 9x or what have you to make sure nothing ever goes down.<p>Like what&#x27;s wrong with on-prem? Lack of diesel generators? We could just have that without AWS. Bare metal datacenter. Counter to most opinions, I think managing a server isn&#x27;t that difficult. I am sort of a semi-professional and prosumer that has no trouble managing servers for years on end with less downtime than a whole fricking datacenter.<p>There is more serious discussion and new revelations around this [1], [2]. Sometimes it is hard to ask questions about layers of abstractions that have built up and no one dares to think about getting rid of them.<p>[1] <a href="https:&#x2F;&#x2F;www.economist.com&#x2F;business&#x2F;2021&#x2F;07&#x2F;03&#x2F;do-the-costs-of-the-cloud-outweigh-the-benefits" rel="nofollow">https:&#x2F;&#x2F;www.economist.com&#x2F;business&#x2F;2021&#x2F;07&#x2F;03&#x2F;do-the-costs-o...</a><p>[2] <a href="https:&#x2F;&#x2F;oxide.computer&#x2F;" rel="nofollow">https:&#x2F;&#x2F;oxide.computer&#x2F;</a>

<a href="https:&#x2F;&#x2F;health.aws.amazon.com&#x2F;health&#x2F;status" rel="nofollow">https:&#x2F;&#x2F;health.aws.amazon.com&#x2F;health&#x2F;status</a> is now showing the issue.<p>Our first canaries fired for this at 9:43 PDT local time. The status page updated at 10:13.

[10:33 AM PDT] [10:33 AM PDT] We are investigating increased error rates for invokes in the US-WEST-2 Region. We do not yet have a root cause, but are investigating multiple potential root causes in parallel. In addition, we are implementing filters on inbound traffic from a set of sources with recent significant traffic shifts, which may help mitigate the impact. We do not yet have a solid ETA, but will continue to provide updates as we progress.<p>[10:13 AM PDT] [10:10 AM PDT] We are investigating increased error rates for invokes in the US-WEST-2 Region.<p>AWS AppSync AWS Batch AWS Certificate Manager AWS Cloud9 AWS CloudShell AWS Device Farm AWS Global Accelerator AWS Greengrass AWS IoT 1-Click AWS IoT Device Management AWS Lambda AWS Proton AWS Resource Access Manager AWS RoboMaker AWS Service Catalog Amazon AppStream 2.0 Amazon CloudWatch Amazon Connect Amazon Elastic Compute Cloud Amazon Elastic Container Registry Amazon Elastic MapReduce Amazon EventBridge Amazon FinSpace Amazon Kendra Amazon Lightsail Amazon Location Service Amazon Managed Workflows for Apache Airflow Amazon Nimble Studio Amazon Pinpoint Amazon SageMaker Amazon WorkSpaces EC2 Image Builder

Yep same symptoms for us as the August outage and support confirmed an outage for us as well

Bunch of<p>Database error: [Amazon](500310) Invalid operation: No response body.<p>Database error: [Amazon](500310) Invalid operation: curlCode: 28, Timeout was reached<p>in our logs during the last hour

My job application at amazon.jobs today got interrupted by this, I think. I hope they don&#x27;t get a malformed application :&#x2F;

From the latest update:<p>&gt; While we have seen improvements in error rates since 10:40 AM PDT, recovery has stalled and we do not have a clear ETA on full recovery. For customers that have dependencies on API Gateway and are experiencing error rates, we do not have any mitigations to recommend to address the issue on the customer side.

I&#x27;m working on an app that tracks downtimes. I put some of my latest data up here: <a href="https:&#x2F;&#x2F;app.awareops.com&#x2F;whatisdown&#x2F;orgs&#x2F;a7f95108-ead0-4718-8940-d61e0d80510a" rel="nofollow">https:&#x2F;&#x2F;app.awareops.com&#x2F;whatisdown&#x2F;orgs&#x2F;a7f95108-ead0-4718-...</a>

Seeing API gateway issues here: <a href="https:&#x2F;&#x2F;www.taloflow.ai&#x2F;is-aws-down&#x2F;us-west-2" rel="nofollow">https:&#x2F;&#x2F;www.taloflow.ai&#x2F;is-aws-down&#x2F;us-west-2</a>

I&#x27;m seeing multiple services as down: ECS, Medialive, etc.

Lambda is returning 502 errors in US-West for me as well.

Must be a specific AZ, not seeing any issues here<p>EDIT: I see now that it&#x27;s an API Gateway issue, so that&#x27;s why my team isn&#x27;t impacted.

Could any of you clarify to me why do we have &quot;Edge&quot; domain rather than a specific region?

Cloud9 is also down. Joy.<p>Edit to add: I can&#x27;t seem to access my NHL Season Tickets via Ticketmaster either...

S3-megalodon on call checking in

We&#x27;re having issues as well

Now I know why cloud shell wasn&#x27;t working around noon PDT

Hey atleast it isnt us-east-2 this time

what happened to stop.lying.cloud?

us-east-1&#x27;s revenge.

3h 52m and counting

my least favorite region tbh

Here here!

Same here