There's Just No Getting Around It: You're Building a Distributed System (2013)

&gt; There are many reasons why an organization would need to build a distributed system, but here are two examples:<p>&gt; - The demands of a consumer Web site&#x2F;API or multitenant enterprise application simply exceed the computing capacity of any one machine.<p>&gt; - An enterprise moves an existing application, such as a three-tier system, onto a cloud service provider in order to save on hardware&#x2F;data-center costs.<p>When you&#x27;ve exhausted the capacity of a single machine, typically you don&#x27;t jump straight to a distributed system. You can scale out horizontally with a stateless application layer, as long as the data storage on the backend can handle all the load. You can also scale database reads horizontally, using read replicas.<p>This horizontal scale-out is not a distributed system, since consensus (&quot;source of truth&quot;) still lives in one machine.<p>So I think a better phrasing of #1 would be &quot;When your write patterns exceed the computing capacity of any one machine&quot;.

Regarding testing distributed systems. Chaos Monkey, like they mention, is awesome, and I also highly recommend getting Kyle to run Jepsen tests. But we still need more tools on this front, so we built <a href="https:&#x2F;&#x2F;github.com&#x2F;gundb&#x2F;panic-server" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;gundb&#x2F;panic-server</a> which integrates with Mocha (and other unit test frameworks) to make it easy to run failure scenarios across real and virtual machines. It has been a life saver for me.

Any time you have something talking to a database, cache, or pretty much anything else you have a distributed system, you just don&#x27;t know it yet.

&gt; Geographies. Will this system be global, or will it run in &quot;silos&quot; per region?<p>Although I&#x27;ve only worked at Amazon for about a year, I&#x27;ve learned that you should always consider building siloed&#x2F;regionalized applications—if not, expect major headaches when the service needs to be deployed in multiple environments.

I&#x27;m going to join ACM just to see what happens. Who even is in this thing?

This is a verbose paper written by a tech bro that over-generalizes &quot;How To Build A Scaling High Availability Web App&quot;, but fails to explain how such systems are designed in general. If you&#x27;re a tech startup and have never worked in this industry before, the very last paragraph is useful.<p>My personal ideal system design is one that you can pick up and drop into a single machine, a LAN, a cloud network, geographically dispersed colocated datacenters, etc without relying on 3rd party service providers. If you go from a start-up to a billion dollar company, you will eventually have offices with their own labs, dev, qa, middleware and ops teams, datacenters and production facilities, and your hardware and software service providers will run the gamut. If you can abstract the individual components of your system so that dependencies can be replaced live without any changes to any other part of the system, you have the start of a decent design.<p>However, nobody I&#x27;ve ever worked for designed their system this way initially, and they made millions to billions of dollars, so there certainly is no requirement that you have a perfect distributed system design for your emoji app start-up.