Have the Tables Turned on NoSQL?

Honestly, I think the more important takeaway isn’t the decline of NoSQL. As the article says, it’s the conclusion that’s what’s good for FAANG isn’t necessarily what’s good for your average project. I know this sounds obvious but it’s been a constant source of frustration for me with the hype train.<p>You could apply the same logic to a whole bunch of tech like Kubernetes, service mesh etc etc and arrive at the same result.<p>Every tech has a trade off, understanding it is critical. Don’t pick tech spot SOLELY based on FAANG use.

&gt; Because an SQL database uses a schema or structure, this means changes are difficult. Say you’re running a production database full of a million records.<p>Articles like this one perpetuate the myths in the minds of young developers. First off, “millions of records” is nothing this days. More importantly, the scheme ends up living somewhere. If it’s not in your database, you’re likely managing it in the app. There’s no free lunch when it comes to scheme for a typical SaaS.

This blog post is calling at least two different things SQL, and it&#x27;s kind of infuriating.<p>SQL as a query language is never going away. Virtually every database has found it necessary to offer a SQL-like query language: Cassandra&#x27;s CQL, HiveQL, Couchbase query language, and so on. SQL is a human-readable, composable formalism for describing data.<p>What&#x27;s gone away is the practice of writing complex, highly linked, normalized database schemas with layers of constraints and foreign key references. That was banished to the land of stagnant enterprise 10 years ago and is not coming back.<p>The last 10-15 years have been an evolution from mostly static, deeply linked, highly structured data to shallow schemas, append-only updates, denormalized data, and stream processing. If you data is a stream of updates, there&#x27;s not as much pressure to roll back. If your data is mostly defined by a series of processing pipelines that live entirely outside your data warehouse, there&#x27;s not a lot of upside on enforcing constraints in the DB. If anything, we have learned that it&#x27;s very useful to offer different denormalized views of the data in the DB to different consumers.<p>MS SQL Server is not roaring back. Businesses have just learned to unbundle data processing from data warehouses. Data warehouses now have fewer tasks to focus on, such as scalability. And if your DB is just a dumb replica with a flat schema, whether it&#x27;s an RDBMS or not is pretty unimportant.

Why did NoSQL become popular? Was it the huge datasets required by the internet?<p>Before SQL, there was already no SQL. I don&#x27;t mean that as a semantic joke, but that databases existed before relational databases that were faster. Relational DB were too slow to even be usable, until B-trees made them barely feasible in performance (and still much slower than previous DB).<p>The advantage was flexibility: you could change the database organization without having to rewrite application. Similarly, if your application needed data in a different form, you could make it seem that the database was already in that form.<p>So SQL was like a glue between systems that could transform the <i>structure</i> of the data - much like high school algebra can put an equation into a different form, that is equivalent but more convenient.<p>I can imagine, that back in 1970&#x27;s, computing power was growing much faster year-by-year, than typical database sizes were. So, although &quot;slow&quot;, they became &quot;fast enough&quot; for more and more use-cases.<p>But in 2010&#x27;s, internet datasets were growing much faster - and computing power wasn&#x27;t. So Relational DBs weren&#x27;t &quot;fast enough&quot; for these cases... hence &quot;NoSQL&quot;.<p>Is that about right?

A while ago I&#x27;d say! Last time I heard anyone seriously excited about NoSQL was several years ago. It still has its place, but it seems like Postgres is the hype these days.

It is fractally amazing to see the exact same false dichotomy within data stores, DBMSes, and query engines themselves playing out in the &quot;market view&quot; of those same products.<p>That is:<p>The tradeoffs between all these systems has always been the effort required to create, modify, and maintain well-groomed (albeit rigid) schemas and data models versus the speed, scale and agility of a schema-on-read &#x2F; &quot;unstructured&quot; data storage mechanism.<p>Which is then counterbalanced by the tradeoff between getting quick, accurate (albeit rigid) answers of a well-managed data warehouse vs. having to string together fragile, complex ad-hoc wrangling and querying code.<p>So pick your poison: a junk drawer full of Legos or a beautiful sculpture with the head and an arm missing.<p>And the obvious answer is for most organizations you need both! Agility for bottoms-up discovery and exploration, and rigidity for top-down hard facts and shared objectives. (Maybe it&#x27;s a lakehouse, maybe it&#x27;s not, TBD.)<p>And then there&#x27;s this meta thing where NoSQL was pitched as a disruptor, agile, low barriers to entry, and RDBMSes and data warehouse vendors got this reputation as slow, rigid, too in love with their creations to change ...<p>And now there&#x27;s this reverse pushback - oh, actually these NoSQL vendors need to grow up and mature their products, that agility was just a lot of hype and chaos, these data warehouse vendors had the right ideas, they&#x27;ve learned to play the NoSQL vendors&#x27; game better than they have and their go-to-market strategies have stood the test of time ..<p>When (again!) the answer is you need both: disruptors bringing different paradigms to market, letting organizations pick and choose capabilities based on their needs, making legacy vendors adapt and evolve.<p>Funny to see that rhyme.

So glad I migrated to SQL recently. I thought I had unstructured data and I had no real need for relational data. But oh boy I was wrong. Want a customer list, billing, emails, linked accounts with those users, etc. All of this was such a pain in mongo and remnants of messy schema still lurk in our codebase. Reminds me alot coding in typed languages like C vs. python or JS. But in the case of mongo I think I was getting the worst of everything ;)

Nah, the pendulum has just swung back in the opposite direction. Give it 10 or 20 years, and just like strongly typed programming languages, NoSQL will be all the rage again, and you&#x27;ll be &quot;an idiot&quot; for not using it, again.

Fashion-oriented posts like this are always a shitfest, and this one is no different. It is setting up a variety of different architectural styles as if they were in competition for the &quot;top spot&quot;, which is to say the only option that should be applied in all use cases. This isn&#x27;t just garbage, it&#x27;s actively encouraging a whole breed of shitty engineers who never learn how to approach solving a problem.<p>&gt; The goal of a NoSQL database, on the other hand, is to ensure ultimate scalability by making sure that the data is stored in a format that can be shared—or sharded—across multiple servers<p>From here, it then proceeds to list architectural specialisms that have absolutely nothing to do with scaleability<p>- &quot;Document stores&quot; excel at managing compound representations of data, they do an amazing job of minimizing IO when many small sets of (usually hierarchically structured) data can be stored as a single unit. Document stores map particularly well to the &quot;REST&quot; service architectures in the original Sam Ruby meaning of the word<p>- Graph databases are (usually but not always) document stores that excel at indexing and executing transitive queries. Their innovation is not in storage, but in querying particular kinds of data sets with complex (and possibly undefined upfront) relationships using queries that are also likely complex and possibly undefined. This has more to do with expressiveness than scaleability<p>- Column stores excel at managing timeseries. Like document stores, their entire point is IO and processing optimizations that become possible when data is in a particular shape -- varying with a particular profile, and with high redundancy when viewed along a single (usually time) axis. Column stores absolutely rock when applied to the right kind of data -- they can provide 20x storage size improvements and similar query execution time improvements. Finally we can say that column stores have something to do with scaleability. A 20x improvement in hardware utilization could very much be make or break for many kinds of common project<p>- Time series databases are column stores.<p>&gt; Because companies like Google and Amazon created these databases for their own massive data stores, the goal was to reduce the time needed to grab a piece of data<p>Every. Single. One of these architectural styles long predates the FAANG-industrial complex.<p>&gt; NoSQL databases don’t offer much in the way of transaction management or real coding<p>Real coding?<p>&gt; NoSQL databases like MongoDB just take data and store it<p>Nobel Prize stuff right here.<p>I stopped reading

This is filled with incomplete information. MongoDB has had transactions since 4.0 and a strong consistency model by default from the start. That&#x27;s not to say they didn&#x27;t have some bad defaults early on... This article makes gross generalizations and just doesn&#x27;t really add all that much value.<p>Document databases offer a full-featured general-purpose alternative and really shouldn&#x27;t be compared to Key&#x2F;Value stores at all. They&#x27;re only being lumped together since both are &quot;nosql&quot;, a fairly tired term at this point

I have a very antagonistic relationship with NoSQL databases because the vast majority of people get nothing out of using one, and yet every resource a newcomer to programming (on the Node.js ecosystem at least) recommends using MongoDB with Mongoose (an ORM for a NoSQL database? Why?), leading them down a path they really have no business walking because they could have instead learned the widely used, time-tested traditional SQL databases.

Each has its place.<p>Three years ago I started a new project and elected MariaDB. I was coming from a project that was using MongoDB. Because the new project seemed to have very structured data, mostly coming from third party systems, I opted for a structured solution.<p>Three years later and my structured database has tons of tables and requires lots of brain twisting joins. It slowly evolved this way, while our UI basically evolved to use a single React &quot;state&quot; to represent an &quot;order&quot;.<p>It&#x27;s tempting to consider what it might be like to store an order as a single Mongo document and forget all this structure.

Interesting blog post. Thanks for sharing. This part, in particular, resonated with me:<p>&gt; Querying data is a little harder. Apache’s Cassandra uses Cassandra Query Language or CQL which, interestingly, does not allow for joins. MongoDB just sends JSON objects in reaction to requests. Need all users in Ohio? MongoDB sends a big chunk of data.<p>I fondly recall the late night debates with fellow colleagues in the industry, several years ago, when we were pitching a database design to a startup bank in South Africa.<p>Back then during those fights some even suggested that the NoSQL vs <i>Yes</i>SQL debate was a religious war - much like vi vs emacs - but in the case of data storage it quickly became obvious that each philosophy had its respective strengths and weaknesses - which were in turn easy to understand, to sell, and to add value with.<p>But nowadays I must confess I do not know of many shops using NoSQL, and I suspect it is for the reasons quoted from the blog post that I shared above.<p>I would love to read your insight if you&#x27;ve been part of a big NoSQL deployment. We struggled to sell it, so I suspect we must have missed out on some interesting opportunities.

One thing that I think is going away is eventual consistency at the application layer. It is too much of a technical debt and error prone for most applications. It is much easier to reason about a consistent database.<p>And systems like Google Spanner, and CockroachDB show that you can have a consistent database with good scaling and good performance.

NoSQL was the last time I ranted about a stupid technology that became popular, nice to see its finally being put to death publicly. I just ignore them now and wait for their inevitable death (node.js I&#x27;m looking at you)<p>Edit: was just thinking the defining characteristic of these sorts of technologies is they are advocated as replacements for things that already exist, and the people advocating them are not experts in the things they are trying to replace. So they don&#x27;t understand the reasons behind why things are done - NoSQL was obvious for any database person - no transactions, no normalisation. Node.js - tries to replace server coding with something vastly inferior, and it sort of works until you need a proper server.<p>Edit2: lol, time will tell

It’s all about trade-offs.<p>Do you need availability or consistency? <a href="https:&#x2F;&#x2F;martin.kleppmann.com&#x2F;2015&#x2F;05&#x2F;11&#x2F;please-stop-calling-databases-cp-or-ap.html" rel="nofollow">https:&#x2F;&#x2F;martin.kleppmann.com&#x2F;2015&#x2F;05&#x2F;11&#x2F;please-stop-calling-...</a><p>I would personally never build anything mission critical using NoSQL, sure it’s fast and easy to use, but it might render unreliable in some situations when it’s most important.<p>However 99% of stuff SWE use to build are nowhere close to that level of importance.<p>As long as you know your tool and requirements feel free to use whatever you want, even JSON file on hard drive.

I don&#x27;t have much experience but you can easily mess up both SQL and NoSQL. I recently picked up Datomic and honestly I&#x27;m not looking forward using SQL again for a while.

NoSQL, I always understand it as Not Only SQL. At our place we use MongoDB (Main store), SQL, Big Query, Redis, ElasticSearch, then we also store data in S3 that we don&#x27;t want to query or have the cost of storing in the DB.<p>Pick the right DB for your requirements. Management of them isn&#x27;t that hard as they&#x27;re hosted solutions, we&#x27;ve only got to deal with the cost of when a version is EOL, so upgrades or when certain queries will no longer work.

I&#x27;m surprised that this article is not even mentioning one NewSQL DB like:<p>- Google Spanner<p>- CockroachDB<p>- TiDB<p>- YugabyteDB<p>- FaunaDB<p>Can it be that those are so little known by most developers?

There are couple factors:<p>1. fast hardware, ie. SSD and a lot of RAM allow nowdays to run classic SQL engines without understanding much how to make them perform. All these &quot;millions&quot; or records fit in RAM these days.<p>2. availability of SQL[-like to various degree] frontends to NoSQL engines.

&gt; First, we have to remember that NoSQL databases are probably great for Amazon and Google but not so great for your side hustle<p>Hold up, what? I hooked up a sign up form with MongoDB Atlas, and it&#x27;s working brilliantly and it&#x27;s pretty much free for my side hustle... and it was almost effortless to learn how to implement it... so I don&#x27;t really agree with this article.<p>Some background: I&#x27;m a UX designer, not a dev, and don&#x27;t have any experience with MySQL (or any free-ish, easy cloud hosted MySQL services, so MDB Atlas was a no-brainer)<p>(recently I&#x27;ve also been experimenting with Fauna and Supabase)

I stopped reading at &quot;Traditional SQL uses related tables connected by IDs&quot; ... that&#x27;s what ends up with databases designed with pointless auto-incremented proxy-primary keys.

Any experience with something like distributed SQL from CockroachDB? The project sounds amazing to be quite frank, but I&#x27;d love to hear from someone with first hand experience.

I just wish that Cloud SQL on Google Cloud would make it easy to do multi region replication seems impossible with Postgres.<p>I have been using a NoSQL solution lately and each time when you have the need to do joins its a pain as it needs to be done manually and the lack of full text search ain’t fun either which required to use some paid full text search solution. Datastore painful

The article says that NoSQL has no relations. Is that the case? I would have assumed that, say, you&#x27;d make a blog system by making user entries with a list of blog post IDs and then each blog post gets its own entry with its data. If not, are you querying and processing a user&#x27;s entire blog post history everytime you make an update?

SQL is a number of different discrete things including:<p>- A query language<p>- A schema definition language<p>- A storage engine<p>- A server<p>You can mix and match the parts so it works for you.

When I read the title I thought: this must be yet another closed question.<p>But this is actually posted on their blog.<p>Too bad that you cannot discuss such topics on Stackoverflow. It’s an interesting topic, and it would cause an useful debate.

I find document stores shine when you have known access patterns and can line your data up to meet those patterns. I find relational stores shine when you have unknown access patterns.

I thought the MySQL and Postgres addition of JSON data types took a little of the wind out of nosql sails.<p>That and having a non-uniform api to access data from each type of nosql database.

After doing this [1], I never looked back tbh.<p>[1] <a href="https:&#x2F;&#x2F;www.sqlteaching.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.sqlteaching.com&#x2F;</a>

Ultimately comes down to the right tool for the job. Larger organizations tend to short list a selection of databases to choose from as new applications create new data requirements. These tend to include a NoSQL option, some combination of legacy databases, a data caching or message broker tool, and an open source relational database. Postgres has a lot of momentum as the &quot;new&quot; relational option.

If only the title was intended as a pun :)

Yup. Told you so.

yes.

This same article could have been published in 2011 with different headings. In fact, it almost seems like they intended to publish an informational article and some editor came in to write some headings likely to inspire some hot takes in response.<p>When was the last time you heard someone seriously say that &quot;NoSQL&quot;—that&#x27;s right, don&#x27;t even name a database or name any characteristic of its operation, how it scales, how it&#x27;s queried, its consistency guarantees, its maintenance overhead—is easily comparable to a vertically-scaling SQL engine? The whole rhetoric of tables turning implies that you&#x27;d choose a database for cultural reasons outside of hiring ability.... who thinks like that?

yes.