US-East AWS Connectivity Issues

This appears to be connectivity issues entirely to/from the Internet or other EC2 regions from a single availability zone in us-east-1. The intra-AZ networks within us-east-1 have remained available during the event. One of the AZs we use was affected, but no external traffic flows to it. I noticed this because an auto-scale group was trying to bring up instances inside of the affected zone (our us-east-1a) and was unable to contact a server outside of AWS.

Did anyone else notice a huge spike in incoming network traffic on their EC2 instance immediately before the outage? Roughly 9:55AM EST.

Does anyone know why in the world they display a green checkmark with a near invisible little 'i' for this?

Amazon is continuing the trend of announcing outages 30 minutes after they start.<p>Just signed up for a support contract since the status page said everything was fine.

We (ScraperWiki) can still access some of our US East servers. From those, can daisy chain SSH into the ones that are offline. Those servers can&#x27;t see the world, but are working fine and can see other EC2 instances.

I wonder if it extends beyond Amazon, since my gmail now doesn&#x27;t pull anything up after 2009, web or IMAP.

I&#x27;m getting external monitoring failures that are firing on and off, but have no problem reaching the servers or the site.<p>Interestingly newrelic is reporting the site down at the same time it is reporting a normal level of load on it.

I&#x27;ve recently moved some of our servers over to Digital Ocean, but I&#x27;m still using AWS for DNS since their Route 53 weighted DNS with health checks work as a basic load balancer for our needs. I&#x27;m seeing DNS health checks that point at individual servers at Digital Ocean that are showing 0.91 for a status (1 being up and 0 being down. The alarms attached to the health checks keep flipping from &quot;alarm&quot; to &quot;ok&quot; and causing tons of alerts. As of about 15 minutes ago all of my checks started holding steady back at a status of 1 (ok) Good stuff :)

ELBs are also having problems. One of mine is reporting all instances out of service (transient error), then all instances in service, intermittently. But the ELB is never reachable (even when it reports all instances healthy and up). All instances behind this one are reachable, up and running. US-East-1.<p>Some of our other instances are reachable but some are not, same as others have been reporting.

Why does this never happen to AWS West? I should really get to migrating over with 3 outages in the past 2 years on US East.

There are definitely issues with network connectivity between AZs as well as public internet connectivity.

I got into one of our machines that presented the connectivity issues from another one which was still reachable. It had no external (curl www.google.com) connectivity. Just two minutes ago it started resolving again.

It looks ok now for us (appfluence.com), but even when it was down, our website was still up, only the sync services went offline. And even then, they were accessible from the web server...

It&#x27;s region us-east-1c for now, at least from where I&#x27;m sitting... I have instances in other us-east datacenters that are fine.

So what is the best way to balance a hosted site between Amazon and a separate service? Because these connectivity issues suck.

this has taken openredis offline: <a href="https://twitter.com/openredis" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;openredis</a><p>heroku is also reporting issues:<p><a href="https://status.heroku.com/incidents/554" rel="nofollow">https:&#x2F;&#x2F;status.heroku.com&#x2F;incidents&#x2F;554</a>

All of mine just started magically working

Aaand were beck up now.

Always on a Friday...

Down in Manhattan