AWS Cognito is having issues and health dashboards are still green

We hired an engineer out of Amazon AWS at a previous company.<p>Whenever one of our cloud services went down, he would go to great lengths to <i>not</i> update our status dashboard. When we finally forced him to update the status page, he would only change it to yellow and write vague updates about how service might be degraded for some customers. He flat out refused to ever admit that the cloud services were down.<p>After some digging, he told us that admitting your services were down was considered a death sentence for your job at his previous team at Amazon. He was so scarred from the experience that he refused to ever take responsibility for outages. Ultimately, we had to put someone else in charge of updating the status page because he just couldn&#x27;t be trusted.<p>FWIW, I have other friends who work on different teams at Amazon who have not had such bad experiences.

Now is probably a good time to plug some of the open source alternatives to vendor locked in identity solutions:<p>- <a href="https:&#x2F;&#x2F;github.com&#x2F;ory" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;ory</a><p>- <a href="https:&#x2F;&#x2F;github.com&#x2F;dexidp&#x2F;dex" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;dexidp&#x2F;dex</a><p>- <a href="https:&#x2F;&#x2F;github.com&#x2F;authelia&#x2F;authelia" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;authelia&#x2F;authelia</a><p>- <a href="https:&#x2F;&#x2F;github.com&#x2F;keycloak&#x2F;keycloak" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;keycloak&#x2F;keycloak</a><p>- <a href="https:&#x2F;&#x2F;www.gluu.org&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.gluu.org&#x2F;</a><p>- <a href="https:&#x2F;&#x2F;github.com&#x2F;accounts-js&#x2F;accounts" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;accounts-js&#x2F;accounts</a>

&gt; This is also causing issues with Amplify, API Gateway, AppStream2, AppSync, Athena, Cloudformation, Cloudtrail, Cloudwatch, Cognito, DynamoDB, IoT Services, Lambda, LEX, Managed BlockChain, S3, Sagemaker, and Workspaces.<p>Well, this is a major outgage

Five hours later and nothing has changed. For a company like Amazon this should be unacceptable.<p>Before someone replies and says use a different AZ, that&#x27;s not possible for everyone. If you use a 3rd party service that is hosted on us-east-1 you can&#x27;t do anything about it. For example, many Heroku services are broken because of this.

&quot;I want to have an AWS region where everything breaks with high frequency...&quot;[0] discussed here [1]<p>[0] <a href="https:&#x2F;&#x2F;twitter.com&#x2F;apgwoz&#x2F;status&#x2F;1292519906433306625?s=20" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;apgwoz&#x2F;status&#x2F;1292519906433306625?s=20</a><p>[1] <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=24103746" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=24103746</a>

Isn&#x27;t it common practice to host your status board on someone else&#x27;s infrastructure?<p>In 2017 there was an S3 issue that supposedly affected their ability to post. I believe they said that they were updating how they posted to the status board so that there would no longer be a dependency on S3. Well, I guess whatever they&#x27;re dependent on now broke.

Large-scale events (LSEs) are becoming more and more common. It&#x27;ll keep getting worse.<p>AWS has to take a hard look at how they build their software. Their bad engineering practices will eventually catch up to them. You can&#x27;t treat AWS the same as Alexa. Sometimes it&#x27;s smarter to take your time to ship stuff instead of putting it out there. Burning out your oncall engineers is not a feasible long-term plan.<p>AWS will be in deep trouble when&#x2F;if GCE fixes their customer support.

Can anyone explain why status pages are so difficult. Theres even statups like status.io dedicated to this one thing.<p>It really does seem that anytime there is an outage more often than not the status page is showing all green traffic lights. Making it redundant as a tool to corroborate whats happening.<p>How did AWS status page compare with status.io&#x2F;aws?

I think we are learning everything that uses AWS Kinesis internally which is cool. It’s always fascinating to learn how AWS works on the backend.

Cognito is one of the most frustrating AWS services I have to work with, it is almost, but not quite, entirely unlike an SP.<p>We&#x27;re using it to federate customer IDPs through user pools, but this ends up with customer configs being region specific.<p>Has anyone figured out how to set up Cognito in multiple regions without the hijinx of having the customer setup trusts for each region? Not to mention, while multiple trusts are I think possible with ADFS (not that I&#x27;ve tested it), I&#x27;m pretty sure that Okta doesn&#x27;t support multiple trusts, so regardless of how many regions, we&#x27;d still be SOL there...

&gt; It&#x27;s not posted on SHD as the issue has impacted our ability to post there.<p>Is that not a massive catch-22 for a service dashboard?

Almost 9 years have passed by and nothing has changed. The dashboards continue to remain green.<p><a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=3707590" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=3707590</a>

&quot;This issue has also affected our ability to post updates to the Service Health Dashboard.&quot;<p>Last sentence of the alert at the top of the page.

I&#x27;m in the UK and this now may have cascaded onto VISA<p><a href="https:&#x2F;&#x2F;downdetector.co.uk&#x2F;status&#x2F;visa&#x2F;map&#x2F;" rel="nofollow">https:&#x2F;&#x2F;downdetector.co.uk&#x2F;status&#x2F;visa&#x2F;map&#x2F;</a><p>I am unable to order my Papa Johns pizza<p><a href="https:&#x2F;&#x2F;imgur.com&#x2F;u5QSszv" rel="nofollow">https:&#x2F;&#x2F;imgur.com&#x2F;u5QSszv</a>

&gt; &quot;This issue has also affected our ability to post updates to the Service Health Dashboard.&quot;<p>This is why I prefer 3rd party monitoring systems to track health of my internal monitoring systems.

Many applications – Including Anchor, Adobe Spark, Flickr, SiriusXM and Roku reported disruption caused by this outage. <a href="https:&#x2F;&#x2F;news.alphastreet.com&#x2F;huge-aws-outage-affects-a-wide-range-of-applications&#x2F;" rel="nofollow">https:&#x2F;&#x2F;news.alphastreet.com&#x2F;huge-aws-outage-affects-a-wide-...</a>

Rule #1 of status pages: never put your status page on the same infrastructure it monitors.

Banner on top of <a href="https:&#x2F;&#x2F;status.aws.amazon.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;status.aws.amazon.com&#x2F;</a> just has an update from 8:36AM PST -- just removed -- even thought it&#x27;s only 7:42AM PST. I guess it&#x27;s really manual firefighting there.

There&#x27;s a lot more going on over there...<p>- 7 cloudfront distributions created today are still in &quot;InProgress&quot;, a few already for more than one hour<p>- The support case I created about it doesn&#x27;t show up in my support portal. Direct link to it does work though

Ah yes. It&#x27;s the annual AWS Thanksgiving Holiday major us-east outage.

7:30 AM PST: We are currently blue on Kinesis, Cognito, IoT Core, EventBridge and CloudWatch given an increase in errors for Kinesis in the US-EAST-1 Region. It&#x27;s not posted on SHD as the issue has impacted our ability to post there. We will update this banner if there continue to be issues with the SHD.<p>Was posted 8 minutes ago.

&gt; 2:43 PM PST Between 5:15 AM and 2:28 PM PST customers experienced increased API failure rates for Cognito User Pools and Identity Pools in the US-EAST-1 Region. This was due to an issue with Kinesis Data Streams. We have implemented a mitigation to this issue. Cognito is now operating normally.<p>Seems like they fixed Cognito while Kinesis and many other services are still broken - presumably somehow removing the dependency on Kinesis? It’ll be really interesting if their post mortem explains this mitigation.

Kinesis seems to be down to me. Everything is melting, it is like they have Chaos Monkey perpetually on in us-east-1

Every time I check the Personal Health Dashboard, the number of issues increases; it&#x27;s currently showing 13 open issues for my account. Cloudwatch logs for the last few hours are unavailable; it appears that the log agent is getting errors when it attempts to upload log events. Metrics are spotty or missing.

Maybe AWS should put their dashboards on GCP

It is now affecting ECS and EKS. Having problems scaling own nodes.

And that confirms it for me: Amazon is officially a Day 2 Company.<p>Happened faster than I thought, but based on reading the comments about people who work(ed) there, this seems cut and dried to me.

CloudWatch is definitely one of those &quot;AWS Primitives&quot; services that side effects others when having problems, something similar happened with DynamoDB some years ago.

just looking at this dashboard, I never realized how many services aws has to offer. I’d hate to be the “aws” guy

&gt; This issue has also affected our ability to post updates to the Service Health Dashboard.<p>This is when you fall back to the Tumblr blog for status updates.<p>&lt;rimshot&gt;

what a scam. who can hold them accountable for cheating those who paid for uptime guarantees?<p>I guess the lawyers of those who paid for uptime guarantees...

Apparently they can&#x27;t update the status page because of the outage. This happened a few years ago with the massive s3 outage.

We are seeing an elevated rate of failures on our service, which depends on AWS Cognito. Tweeted an update on it: <a href="https:&#x2F;&#x2F;twitter.com&#x2F;outklip&#x2F;status&#x2F;1331705524396625924" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;outklip&#x2F;status&#x2F;1331705524396625924</a>

As of 2020-11-25T17:21Z this is also causing a Heroku outage preventing new spin-ups, which presumably uses these APIs to verify instance health. <a href="https:&#x2F;&#x2F;status.heroku.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;status.heroku.com&#x2F;</a>

friends don’t let friends use us-east-1.

Title should be changed, this is a widespread AWS issue, it’s not specific to Cognito.

This is what&#x27;s called a SNAFU.

Experiencing 504&#x27;s from Cognito too, our users can&#x27;t log in.<p>&quot;amazon-cognito-identity-js&quot;: &quot;^3.2.2&quot; &quot;aws-amplify&quot;: &quot;^2.2.2&quot;

I&#x27;m getting tired of that bullshit. Just admit it.

Mediaconvert just stopped processing our queues two hours ago, in all our accounts. Anybody else is having it? It&#x27;s green on the status board.

Same issue with AWS lambdas I got a: Received malformed response from transform AWS::Serverless-2016-10-31.<p>It is reported now in their service health dashboard.

We&#x27;ve been having lots of issues with Vercel today, since it uses AWS under the hood I&#x27;m guessing that&#x27;s related...

The way ahead should be an independent entity who audits systems and has responsibility to certify that the dashboard represents a true and accurate view of the actual status. Like is done so effectively with company financials.<p>Oh, wait! EY, PWC, and who can forget Arthur Andersen!<p>But, naturally, technology people can solve this better than anyone else, right?

All my CloudWatch alerts are firing &quot;OK&quot; transitions, and AWS ES isn&#x27;t displaying any known instances

We experienced 504 errors from Cognito but seems to be that other services are affected as well

Is anyone else getting &quot;Capacity unavailable&quot; when trying to add tasks in Fargate?

EventBridge has been struggling for about the past 14 hours as well, which means Cloudwatch Events is not too happy; and, I have the impression CWE underpins a surprising diversity of other things at AWS.

Would this explain the washingtonpost.com outage? That site has been displaying a &quot;Welcome to OpenResty!&quot; page for the past 20 minutes or so.<p>EDIT: nevermind, the Post is back, and Kinesis is still erroring.

Also this thread <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=25209508" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=25209508</a>

AWS Status website is down for me.<p>Is there a status website for AWS Status?

I’m trying to find a doc on running cognito using multiple zones and I can’t find much. Anyone have a multi-az cognito deployment running right now?

Any tips on how to collect on SLA credits from this?

Is this the reason I have seen connection errors in duolingo,<p>&gt; upstream connect error or disconnect&#x2F;reset before headers. reset reason: overflow

We&#x27;re getting 504 for our well-known jwks file<p>And request timeouts against cognito-idp.us-east-1.amazonaws.com<p>And the cognito console won&#x27;t load

503s from CloudWatch for us.

They should rename that region to us-chaostesting-1 . Problem solved.

i tried reaching out to amazon support, apparently they are also seeing issues internally and there is a high possibility that these two are related..<p>Their ETA, 2 hours, and then try contacting again!

Is this <i>only</i> affecting us-east-1 or other regions as well?

Having Lambda issues too

There is no problem here. <i>jedi mind trick hand wave</i>

Paddle checkout is down as well (connected to this outage): <a href="https:&#x2F;&#x2F;twitter.com&#x2F;PaddleHQ&#x2F;status&#x2F;1331659286649466881" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;PaddleHQ&#x2F;status&#x2F;1331659286649466881</a>