How to sleep at night having a cloud service: common architecture do's

One thing missing here is to avoid synchronous communication. Sync comms tie client state to server state; if the server fails, the client will be responsible for handling it.<p>If you use queue-based services your clients can &#x27;fire and forget&#x27;, and then your error handling logic can be encapsulated by the queue&#x2F; consumers.<p>This means that if you deploy broken code rather than a cascading failure across all of your systems you just have a queue backup. Queue backups are also really easy to monitor, and make a great smoke-signal alert.<p>The other way to go, for sync comms, would be circuit breakers.<p>My current project uses queue-based communications exclusively and it&#x27;s great. I have retry-queues, which use over-provisioned compute, and a dead-letter for manually investigating messages that caused persistent failures.<p>Isolation of state is probably the #1 suggestion I have for building scalable, resilient, self-healing services.<p>100% agree with and would echo the content in the article, otherwise.<p>edit: Also, idempotency. It&#x27;s worth taking the time to write idempotent services.

I like guides like this that can help beginners bridge the gap between hobby and professional quality development.<p>I’ll add one more tip, the one I think has saved me more sleep and prevented more headache than any other as I’ve developed a SaaS app over the last 5 years.<p>It’s simple: Handle failure cases in your code, and write software that has some ability to heal itself.<p>Here are a few things I’ve developed that have saved my butt over the years:<p>1) An application that is deployed alongside the primary application, tails error logs and replays failed requests. (Idempotent requests make this possible)<p>2) many built-in health checks like checking back pressure on queues and auto-throttling event emitters when queues get backed up<p>3) Local event buffering to deal with latency spikes in things like SQS.<p>I hope to eventually write more about these systems on our blog but I never seem to find the time

You know the one thing that has helped me out the most, an error reporting service AND then addressing _every_ error.<p>That is to say, my service should emit zero 500 errors.<p>Then my reporting is easy to interpret and consistently meaningful. I don&#x27;t have to worry about bullshit noise &quot;oh that&#x27;s just X it does that sometimes.&quot;<p>Sleeping at night is a lot easier when you have less keeping you awake.

Just about everything mentioned here is well-handled by Google App Engine. I still think it’s the way to go for most projects, but I don’t think they’ve marketed themselves well lately. I’m sure there are other good providers too; I don’t see the downside to using PAAS.

&gt; A 4 9’s means you can only have 6 minutes down a year.<p>4 9&#x27;s is 52 minutes of downtime a year. Keep in mind that single region EC2 SLA is only 99.99%. And if you rely on a host of services with an SLA of 99.99, yours is actually worse than 99.99. So if you want to actually get to 99.99, your components have to be better than this, meaning you will have to go multi-region. So achieving this is actually way harder than this simple step.

I&#x27;d recommend using an APM product off the shelf to get a lot of the mentioned functionality in the article (Monitoring, Tracing, Anomaly Detection). I would definitely _not_ recommend trying to roll all that yourself, unless you have a ton of time and resources.<p>There&#x27;s a few good ones out there, we use Instana and it&#x27;s working really well.

This is all good advice for the app tier, but in my experience the most painful outages relate to the data store. Understand your read&#x2F;write volume, have a plan for scaling up&#x2F;out, implement caching wherever practical, and have backups.

Good article. I would add one thing to this - pick a database that scales horizontally and is distributed. CockroachDB, Elasticsearch, Mongo, Cassandra&#x2F;Scylla are all good choices. If you lose one node, you don&#x27;t have to be afraid of your cluster going down, meaning you can do maintenance and reconfiguration without downtime. If your load is low or bursty you can even get away with running these on some small servers such as t3 (probably minimally t3.larges). Running a cloud managed database is also a good option.

Having only one mirror is scary. If one goes down, its like murphy&#x27;s law kicks in. So you want at least 3 things to go wrong in order to take down your system, 2 is not enough. Also have redundancy <i>everywhere</i> if your checker agent stops working for example. You want 2 of everything and at least 3 of those that should never fail.

As a solo founder, I have almost everything mentioned in this article set up, except CI&#x2F;CD. I can certainly see its value, but being able to easily take down parts of my production system and replace them with instrumented variants is very useful to me when things go wrong. I find that CI usually gets in the way of this. Maybe it&#x27;s just a bad habit that I need to ditch :)

Haven’t seen anything related to third-parties service that your cloud service relies on. I’m talking mostly about APIs that you can use that might crash at some time. Any recommendations on that part?

This is a great list. I feel a little happy with myself that I knew about most of these.<p>Except for identifying each request, I had never heard about that. It&#x27;s so simple yet so brilliant, gotta start doing it.

Good article. Fire drills are worthy of mention. Simulate parts going down, practice recovery.

Nice article, may I ask what tools you used to produce the illustrations?

Pretty funny that HN traffic seems to have killed the site.

Great article Sada :). Hope OneNote is treating you well!

I applaud the author for sharing their notes. But also, this is why HN (and general upvote-anything-that-looks-interesting forums) sucks. If you are actually defining architecture, you should not be reading these kind of blog posts. I get that they are interesting to the layman, but so is The Anarchist&#x27;s Cookbook. <i>Don&#x27;t make whatever you read in The Anarchist&#x27;s Cookbook</i>.<p>And I&#x27;m crabbing about this because <i>I am easily susceptible to Anarchists Cookbooks</i>. I have had to implement X tech before, and googled for &quot;How do I X&quot;, and some blog post came up saying &quot;For X, Use Y&quot;. I&#x27;m too lazy to read 5 books on the general concept, so I just dive in and immediately download Y and run through the quick-start guide. After spending a while getting it going and getting past the &quot;quick start&quot;, I wonder, &quot;Ok, where&#x27;s the long-start? What&#x27;s next?&quot; And that doesn&#x27;t exist. And later, after a lot of digging, it turns out Y actually really sucks. But the blog post didn&#x27;t go into that. I wasted my time (my own fault) because I read a short blog post.<p>A lot of people live by <i>Infrastructure as Code</i>, and so they will reach for literally anything which has that phrase in its description. But you don&#x27;t need it to throw together an MVP, and a lot of the IaC &quot;solutions&quot; out there are annoying pieces of crap. I guarantee you that if you pick any of them up, you are in for months of occasionally painful edge cases where the answer to your problem is <i>&quot;You just weren&#x27;t using it the right way.&quot;</i><p>In reality, if you want to be DevOps (yes, I&#x27;m using DevOps as an adjective, ugh) you should probably develop your entire development and deployment workflows by hand, and only when you&#x27;ve accomplished all of the basic requirements of a production service by hand (bootstrapping, configuration, provisioning, testing, deployment, security, metrics, logging, alerts, backup&#x2F;restore, networking, scalability, load testing, continuous integration, immutable infrastructure &amp; deployments, version-controlled configuration, documentation, etc), <i>then</i> you can start automating it all. If you&#x27;ve done all of these things before, automating it all from the start may be a breeze. If you haven&#x27;t, you may spend a ton of time on automation, only later to learn that the above need to be changed, requiring rework of the automation.