The DynamoDB Paper

Rick Houlihan did a talk a few years ago about designing the data later for an application using dynamodb. The most common reaction I get from people I show it to- most of them Amazon SDEs who operate services that use Dynamodb- is &quot;Holy shit what is this wizardry?!&quot;<p><a href="https:&#x2F;&#x2F;youtu.be&#x2F;HaEPXoXVf2k" rel="nofollow">https:&#x2F;&#x2F;youtu.be&#x2F;HaEPXoXVf2k</a><p>One of the biggest mistakes people make with dynamo is thinking that it&#x27;s just a relational database with no relations. It&#x27;s not.<p>It&#x27;s an incredible system, but it requires a lot of deep knowledge to get the full benefits, and it requires you, often, to design your data layer very well up-front. I actually don&#x27;t recommend using it for a system that hasn&#x27;t mostly stabilized in design.<p>But when used right, it&#x27;s an incredibly performant beast of a data store.

I&#x27;be been working with DynamoDB daily for a few years now, and whilst I like working with it and the specific scenario it solves for us, I&#x27;d still urge anyone thinking about using it to carefully reconsider whether their problem is truly unique enough that a traditional RDBMS couldn&#x27;t handle it with some tuning. Theycan be unbelievably performant and give so much stuff for free.<p>Designing application specifically for DynamoDB will take _a lot_ of time and effort. I think we could have saved almost a third of our entire development time had we used more of the boring stuff.

&gt; From the paper [0]: <i>DynamoDB consists of tens of microservices.</i><p>Ha! For folks who think two-pizza teams mean 100s of microservices... this is probably the second most scaled-out storage service at AWS (behind S3?), and it runs <i>tens</i> of microservices (pretty sure these aren&#x27;t <i>micro</i> the way most folks would presume &#x27;em to be).<p>&gt; <i>What&#x27;s exciting for me about this paper is that it covers DynamoDB&#x27;s journey...</i><p>Assuming these comments are true [1][2], in a classic Amazon fashion [3], the paper fails to acknowledge a FOSS database (once?) underneath it: MySQL&#x2F;InnoDB (and references it as B-Tree instead).<p>[0] <a href="https:&#x2F;&#x2F;web.archive.org&#x2F;web&#x2F;20220712155558&#x2F;https:&#x2F;&#x2F;www.usenix.org&#x2F;system&#x2F;files&#x2F;atc22-vig.pdf" rel="nofollow">https:&#x2F;&#x2F;web.archive.org&#x2F;web&#x2F;20220712155558&#x2F;https:&#x2F;&#x2F;www.useni...</a><p>[1] <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=13173927" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=13173927</a><p>[2] <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=18871854" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=18871854</a><p>[3] <a href="https:&#x2F;&#x2F;archive.is&#x2F;T1ZNJ" rel="nofollow">https:&#x2F;&#x2F;archive.is&#x2F;T1ZNJ</a>

I&#x27;ve found DDB to be exceptional for use cases where eventual consistency is OK and you have a few well defined query patterns. This is a large number of use cases so it&#x27;s not too limiting. As the number of query patterns grow, indices grow, and costs grow (or pray for your soul you attempt to use DDB transactions to write multiple keys to support differing query patterns). If you need strong consistency, your cost and latency also increases.<p>Oh, and I&#x27;d avoid DAX. Write your own cache layer. The query cache vs. item cache separation[1] in DAX is a giant footgun. It&#x27;s also very under supported. There still isn&#x27;t a DAX client for AWS SDK v2 in Go for example[2].<p>1 - <a href="https:&#x2F;&#x2F;docs.aws.amazon.com&#x2F;amazondynamodb&#x2F;latest&#x2F;developerguide&#x2F;DAX.consistency.html#DAX.consistency.query-cache" rel="nofollow">https:&#x2F;&#x2F;docs.aws.amazon.com&#x2F;amazondynamodb&#x2F;latest&#x2F;developerg...</a><p>2 - <a href="https:&#x2F;&#x2F;github.com&#x2F;aws&#x2F;aws-dax-go&#x2F;issues&#x2F;2" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;aws&#x2F;aws-dax-go&#x2F;issues&#x2F;2</a>

The way that I learnt the ins and outs of DynamoDB (and there is a lot to learn if you want to use it effectively) is by implementing all the Redis data structures and commands on it. That helped understand both systems in one shot.<p>The key concept in Dynamo is that you use a partition key on all your bits of data (my mental model is that you get one server per partition) and you then can arrange data using a sort key in that partition. You can then range&#x2F;inequality query over the sort keys. That’s the gist of it.<p>The power and scalability comes from the fact that each partition can be individually allocated and scaled, so as long as you spread over partitions you have practically no limits.<p>And you can do quite a bit with that sort key range&#x2F;inequality thing. I was pleasantly surprised by how much of Redis I could implement: <a href="https:&#x2F;&#x2F;github.com&#x2F;dbProjectRED&#x2F;redimo.go" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;dbProjectRED&#x2F;redimo.go</a>

Nice write-up from Marc. This definitely hits on the most common problems distributed systems face. I haven&#x27;t read the paper yet but it <i>is</i> pretty cool they published this and talk about changes over time.<p>1. Managing &#x27;heat&#x27; in the system (or assuming that you&#x27;ll have an uniform distribution of requests)<p>2. Recovering a distributed system from a cold state and what that implies for your caches.<p>3. The obvious one that people that do this type of thing spend a lot of time thinking about: CAP theorem shenanigans and using Paxos.<p>Reminds me of the Grugbrained developer on microservices: <a href="https:&#x2F;&#x2F;grugbrain.dev&#x2F;#grug-on-microservices" rel="nofollow">https:&#x2F;&#x2F;grugbrain.dev&#x2F;#grug-on-microservices</a><p>Good luck getting every piece working on the first major recovery. My 100% unscientific hunch is that most folks aren&#x27;t testing their cold state recovery from a big failure, much how folks don&#x27;t test their database restoration solutions (or historically haven&#x27;t).

These days I’d probable take a closer look at spanner. It is a consistent and scalable db. It makes life much easier for developers.<p>Like Cassandra, dynamodb requires the data model to be designed very carefully to be able to get the max out of them.<p>More often than not, that simply adds more complexity; people often underestimate how much a sharded mysql&#x2F;Postgres can scale.<p>My default choice for the longest time: Postgres for the data I care about, ES as secondary index and S3 as blob storage.

One big benefit of DynamoDB over RDS on AWS is that the access layer is API based so you don’t have issues with held open connections when accessing via AWS Lambda.

An underrated part of DynamoDB are its streams. You can subscribe to changes and reliably process those in a distributed way. If you&#x27;re comfortable with the terms &quot;at-least once delivery&quot; and &quot;eventual consistency&quot;, you can build some truly amazing systems by letting events propagate reactively through your system, never touching a data store or messaging broker other than DynamoDB itself.<p>It&#x27;s not for everyone, but when you get a team up and running with it, it can be shockingly powerful.

DynamoDB is (edit: can be) <i>extremely</i> expensive compared to alternatives (e.g. self hosted SQL).<p>Make sure the benifits (performance, managed, scale) outweigh the costs!

A word of caution. The default limit for number of tables per AWS account, for DynamoDB is 2500.<p>Tables are a scarce resource and you want to use single table designs for each app.<p>The design of tables with DDB is fascinating. Once you understand the PK &#x2F; SK &#x2F; GSI dance, design becomes so intuitive.

I wonder how&#x27;s Cassandra doing? I heard companies are migrating away from it.

I&#x27;d like to learn more about their MemDS. Afaik nothing has been made public.

How well does DyanmoDB scale when paired with AppSync and GraphQL? The selling point here being you can use GQL as your schema for the DB too and get automatic APIs for free

Good job! But I&#x27;m wondering when Amazon can start to contribute to open source world...