I want to have an AWS region where everything breaks with high frequency

For those saying &quot;Chaos Engineering&quot;, first off, the poster is well aware of Chaos Engineering. He&#x27;s an AWS Hero and the founder of Tarsnap.<p>Secondly, this would help make CE better. I actually asked Amazon for an API to do this ten years ago when I was working on Chaos Monkey.<p>I asked for an API to do a hard power off of an instance. To this day, you can only do a graceful power off. I want to know what happens when the instance just goes away.<p>I also asked for an API to slow down networking, set a random packet drop rate, EBS failures, etc. All of these things can be simulated with software, but it&#x27;s still not exactly the same as when it happens outside the OS.<p>Basically I want an API where I can torture an EC2 instance to see what happens to it, for science!

Isn’t us-east-1 exactly that?<p>All jokes aside, I actually asked my google cloud rep about stuff like this; they came back with some solutions but often the problem with that is, what kind of failure condition are you hoping for?<p>Zonal outage (networking)? Hypervisor outage? Storage outage?<p>Unless it’s something like s3 giving high error rates then most things can actually be done manually. (And this was the advice I got back because faulting the entire set of apis and tools in unique and interesting ways is quite impossible)

[Disclaimer: I work as a software engineer at Amazon (opinions my own, obvs)]<p>The chaos aspect of this would certainly increase the evolutionary pressure on your systems to get better. You would need really good visibility into what exactly was going on at the time your stuff fell over, so you could know what combination(s) to guard against next time. But there is definitely a class of problems this would help you discover and solve.<p>The problem with the testing aspect, though, is that test failures are most helpful when they&#x27;re deterministic. If you could dictate the type, number, and sequence of specific failures, then write tests (and corresponding code) that help make your system resilient to that combination, that would definitely be useful. It seems like &quot;us-fail-1&quot; would be more helpful for organic discovery of failure conditions, less so for the testing of specific conditions.

When I worked at Skype &#x2F; Microsoft and Azure was quite young, the Data team next to me had a close relationship with one of the Azure groups who were building new data centers.<p>The Azure group would ask them to send large loads of data their way, so they could get some &quot;real&quot; load on the servers. There would be issues at the infra level, and the team had to detect this and respond to it. In return, the data team would also ask the Azure folks to just unpug a few machines - power them off, take out network cables - helping them test what happens.<p>Unfortunately, this was a one-off, and once the data center was stable, the team lost this kind of &quot;insider&quot; connection.<p>Howerver, as a fun fact, at Skype, we could use Azure for free for about a year - every dev in the office, for work purposes (including work pet projects). We spun up way too many instances during time, as you&#x27;d expect, and only came around to turning them off when Azure changed billing to charge 10% of the &quot;regular&quot; pricing for internal customers.

It sounds to me what some people would like is for a magical box they can throw their infrastructure into that will automatically shit test all the things that could potentially go wrong for them. This is poor engineering. Arbitrary, contrived error conditions do not constitute a rational test fixture. If you are not already aware of where failures might arise in your application and how to explicitly probe those areas, you are gambling at best. Not all errors are going to generate stack traces, and not all errors are going to be detectable by your users. What you would consider an error condition for one application may be a completely acceptable outcome for another.<p>This is the reliability engineering equivalent of building a data warehouse when you don&#x27;t know what sorts of reports you want to run or how the data will generally be used after you collect it.

I don&#x27;t see a us-fail-1 region being set up for a number of reasons.<p>One, this is not how AWS regions are designed to work. What they&#x27;re thinking of is a virtual region with none of its own datacenters, but AWS has internal assumptions about what a region is that are baked into their codebase. I think it would be a massive undertaking to simulate a region like this.<p>(I don&#x27;t think a fail AZ would work either, arguably it&#x27;d be worse because all the code that automatically enumerates AZs would have to skip it, which is going to be all over the place.)<p>Two, set up a region with deliberate problems, and idiots will run their production workload in it. It doesn&#x27;t matter how many banners and disclaimers you set up on the console, they&#x27;ll click past them.<p>When customer support points out they shouldn&#x27;t be doing this, the idiot screams at them, &quot;but my whole business is down! You have to DO something!&quot; This would be a small number of customers, but the support guys get all of them.<p>Three, AWS services depend on other AWS services. There are dozens of AWS services, each like little companies with varying levels of maturity. They ought to design all their stuff to gracefully respond to outages, but they have business priorities and many services won&#x27;t want to set up in us-fail-1. When a region adds special constraints, it has a high likelihood of being a neglected region like GovCloud.

I don&#x27;t work with the group directly, but one group at our company has set up Gremlin, and the breadth and depth of outages Gremlin can cause is pretty impressive. Chaos Testing FTW.

Along the same vein, instead of the typical &quot;debug&quot; and &quot;release&quot; configurations in compilers, I&#x27;d love it if there was also an &quot;evil&quot; configuration.<p>The evil configuration should randomise anything that isn&#x27;t specified. No string comparison type selected? You get Turkish. All I&#x2F;O and networking operations fail randomly. Any exception that can be thrown, is, at some small rate.<p>Or to take things to the next level, I&#x27;d love it if every language had an interpreted mode similar to Rust&#x27;s MIR interpreter. This would tag memory with types, validate alignment requirements, enforce the weakest memory model (e.g.: ARM rules even when running on Intel), etc...

A zone not only of sight and sound, but of CPU faults and RAM errors, cache inconsistency and microcode bugs. A zone of the pit of prod&#x27;s fears and the peak of test&#x27;s paranoia. Look, up ahead: Your root is now read-only and your page cache has been mapped to &#x2F;dev&#x2F;null! You&#x27;re in the Unavailability Zone!

That region is called Microsoft Azure. It will even break the control UI with high frequency.

I imagine AWS and other clouds have a staging&#x2F;simulation environment for testing their own services. I seem to recall them discussing that for VPC during re:Invent or something.<p>I&#x27;m on the fence though if I would want a separate region for this with various random failures. I think I&#x27;d be more interested in being able to inject faults&#x2F;latencies&#x2F;degradation in existing regions, and when I want them to happen for more control and ability to verify any fixes.<p>Would be interesting to see how they price it as well. High per-API cost depending on the service being affected, combined with a duration. Eg, make these EBS volumes 50% slower for the next 5min.<p>Then after or in tandem with the API pieces, release their own hosted Chaos Monkey type service.

Show HN! Introducing my new SPaaS:<p>Unreliability.io - Shitty Performance as a Service.<p>We hook your accounting software up to api.unreliability.io and when a client account becomes delinquent, our platform instantly migrates their entire stack into the us-fail-1 region. Automatically migrates back again within 10 working days after full payment has cleared - guaranteed downtime of no less than 4 hours during migration back to production region. Register now for a 30 day Free Trial!

I want this at the programming language level too. If a function call can fail, I want to set a flag and have it (randomly?) fail. I hacked my way around this by adding in some wrapper that would if random, err for a bunch of critical functions. It was great for working through a ton of race conditions in golang with channels, and remote connections, etc. But hacking it in manually was annoying and not something I&#x27;d want to commit.

Failing individual computes isn&#x27;t hard, some chaos script to kill VMs is enough. Worst are situations when things seem to be up but not acceptable: abnormal network latency, random packet drops, random but repeatable service errors, lagging eventual consistency. Not even mentioning any hardware woes.

While these are not exclusive, personally I&#x27;d look instead into studying my system&#x27;s reliability in a way that is independent of a cloud provider, or even of performing any side-effectful testing at all.<p>There&#x27;s extensive research and works on all things resilience. One could say: if one build a system that is proven to be theoretically resilient, that model should extrapolate to real-world resilience.<p>This approach is probably intimately related with pure-functional programming, which I feel has been not explored enough in this area.

There are multiple methods for automating AWS EC2 instance recovery for instances in the &quot;system status check failed&quot; or &quot;scheduled for retirement event&quot; cases.<p>Yet to figure out how to test any of those cloudwatch alerts&#x2F;rules. I&#x27;ve had them deployed in my dev&#x2F;test environments for months now, after having to manually deal with a handful of them in a short time period. They&#x27;ve yet to trigger once since.<p>Umbrellas when it&#x27;s raining etc.

Wait. I thought this was ap-southeast-2

Whichever region Quora is using.

Why does Twitter often fail to load when I open a thread and if I refresh it works. Does Twitter use us-fail-1?

I think people overestimate the importance of failures of the underlying cloud platform. One of the most surprising lessons of the last 5 years at my company has been how rarely single points of failure actually <i>fail</i>. A simple load-balanced group of EC2 instances, pointed at a single RDS Postgres database, is astonishingly reliable. If you get fancy and build a multi-master system, you can easily end up creating more downtime than you prevent when your own failover&#x2F;recovery system runs amok.

At my job, my team owns a service that generally has great uptime. Dependent teams&#x2F;services have gotten into the habit of assuming that our service will be 100% available which is problematic because it&#x27;s obviously not. That false assumption has caused several minor incidents unfortunately.<p>There have been some talk internally of doing chaos engineering to help improve the reliability of our company&#x27;s products as a whole. Unfortunately, the most easily simulatable failure scenarios (e.g. entire containers go down at once instantly, etc.) tend to be the least helpful since my team designed the service to tolerate those kinds of easily modelable situations.<p>The more subtle&#x2F;complex&#x2F;interesting failure conditions are far harder to recognize and simulate (e.g. all containers hosted on one particular node experience 10s latencies on all network traffic, stale DNS entries, broken service discovery, etc.).

You can just do this yourself. Google breaks it&#x27;s systems intentionally for a week every year, it&#x27;s called DiRT week. DiRT takes weeks of planning before people even start debugging.<p>Doing this constantly for a all products in a single region would be absolutely exhausting for SRE teams<p>(Discliamer: I work for ^GOOG and my opinions are my own)

Friendly reminder that for any given single availability zone, the SLA that AWS provides is one single nine. That means that they expect that availability zone to fail 10% of the time, or 6 minutes every hour. this very high failure rate comes up solutely for free, no need for a special region. Therefore, implementing a cross availability zone application that logs when packets are dropped should give you some idea of how your application handles failure.

Relevant snippet in the Google SRE Book:<p><a href="https:&#x2F;&#x2F;landing.google.com&#x2F;sre&#x2F;sre-book&#x2F;chapters&#x2F;service-level-objectives&#x2F;#xref_risk-management_global-chubby-planned-outage" rel="nofollow">https:&#x2F;&#x2F;landing.google.com&#x2F;sre&#x2F;sre-book&#x2F;chapters&#x2F;service-lev...</a><p>Google introduced exactly this to one of their internal services so that downstream dependencies can&#x27;t rely on its extremely high availability.

Toxiproxy [1] is a tool that allows to create network tunnels with random network problems like high latency, packet drops or slicing, timeouts, etc.<p>Setting it up requires some effort (you can&#x27;t just choose a region in your AWS config), but it&#x27;s available now and can be integrated with tests.<p>[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;Shopify&#x2F;toxiproxy" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;Shopify&#x2F;toxiproxy</a>

No one seems to talk about the pricing aspect of this. Developers would want these us-fail-1 regions to be cheap or free since they wouldn&#x27;t be using this for production purposes. And before you know it, a lot of hobbyist developers will start using these as their production setup since they wouldn&#x27;t mind a 1% downtime if they could pay less for it.

Simply host on Google Cloud! They will terminate your access for something random, like someone said your name on YouTube while doing something bad! They don&#x27;t have a number you can call, and their support is ran but the stupidest of all AI algorithms.

There&#x27;s an easier way: spot instances (and us-east-1 as mentioned)<p>As for things like EBS failing, or dropping packages, it&#x27;s a bit tricky as some things might break at the OS levels<p>And given sufficient failures, you can&#x27;t swim anymore, you&#x27;ll just sink.

Sometimes during development instead of checking the return code I&#x27;ll check rand() % 3 or something similar. I&#x27;ll run through the code several times in a loop and run through a lot of the failure modes very quickly this way.

Sort of counter-intuitive, but for small projects, you want resilient hardware systems as much as possible... the larger your scale out, the less reliable you would want them to force that out of hardware into resilient software.

This is such a clever idea. I wonder if amazon are smart enough to actually do this.

Just deploy a new region with no ops support, it&#x27;ll quickly become that.

A great idea! I&#x27;d love to run stuff in this zone. Rotate through a bunch of errors, unavailability, latency spikes, power outages etc every day, make it a 12 hour torture test cycle.

I can see a use case for this being implemented on top of Kubernetes. I&#x27;ve no idea if that&#x27;s achievable, but could go some way to make your code more resilient.

I believe this is available as a service called “Softlayer”

it&#x27;s us-west-1! :D<p>we&#x27;ve had a ton of instances fail at once because they had some kind of rack-level failure and a bunch of our EC2s ended up in the same rack. :(

This would require AWS to invest in Chaos Monkeys.

I believe this is a service called “SoftLayer”

That region should have `us-wtf-1` as code.

Isn&#x27;t this what Gremlin does?

Sounds useful. Crank it up to 99% failure and it becomes interesting science.

Try us-east-1 :)

This is called chaos engineering and many companies built tooling to do exactly this. Netflix pioneered&#x2F;proselytized it years ago. Since you likely don&#x27;t just rely upon AWS services if your app is in AWS, you want something either on your servers themselves or built into whatever low level HTTP wrapper you use. Use that library to do fault injection like high latency, errors, timeouts, etc.

brilliant!

us-east-1?

You can use IBM cloud for that purpose

This is a really great idea.

This is what chaos monkey does.