SQL is 43 years old – Here’s why we still use it today

Been programming for 30+ years, and 99% of my projects use SQL databases. I&#x27;ve tried and dropped NoSQL many times. I still wake up in a cold sweat thinking about the earliest version of Firebase where I had a project that tried to join three tables together to get some meaningful data.<p>I still remember the response of a Firebase team member to my forum question about it - &quot;These days, storage and computing power is cheap - just duplicate the child table as nodes in the main table and do it all that way for every parent&#x2F;child relationship you have. Don&#x27;t worry that you have to duplicate the same set of data for every parent that is related to the same child...That&#x27;s how NoSQL works...&quot; &lt;shudder&gt;<p>Even though I use ORMs in my project these days, every time I have to test a complex query, I write it in raw SQL first and check it before trying to make the ORM duplicate the same query.<p>Granted, NoSQL has its place and its advantages, but for me, when it comes to &quot;money code&quot;, I will stick to SQL.

SQL has the great side effect of creating structures that future employees can understand. Its a set of tables with relationships. Given these you can quickly inspect the structures <i>independent</i> of the programming language dejour. With a few commands a new employee can understand: business logic, hr, billing, reporting, and other major backbone principals in a matter of hours. On the other hand, I have been at companies that jumped on the noSQL &#x2F; ORM &#x2F; wrap it until you can&#x27;t wrap it &#x2F; hide sql (rails). When new employees show up... well... its a bunch of semi&#x2F;no structured stuff spread across thousands of lines of specific logic.

I&#x27;ve already posted on HN how I use SQL in my public affairs data journalism class [0]. To me, there is no better, in terms of accessibility and return on investment gateway language to the power of computation and programming than SQL, with the exception of spreadsheets and formulas. Even if you don&#x27;t go further into programming, SQL provides the best way for describing what we always need to do with data for journalistic purposes -- joining records, filtering, sorting, and aggregating. Ironically, I learned SQL late in my programming career, and initially thought its declarative paradigm to be mysterious and inferior to procedural languages. In fact, I don&#x27;t know how to do anything in SQL beyond declarative SELECT queries (and a handful of database create&#x2F;update admin queries). Turns out this is just powerful enough for me for most app dev work (Rails, Django), and the simplicity is a boon for non-programmers.<p>ProPublica just published a bunch of data-related jobs and positions. The phrase &quot;Proficiency in SQL is a must&quot; makes an appearance: <a href="https:&#x2F;&#x2F;www.propublica.org&#x2F;atpropublica&#x2F;item&#x2F;propublica-is-hiring-a-data-fellow-2017" rel="nofollow">https:&#x2F;&#x2F;www.propublica.org&#x2F;atpropublica&#x2F;item&#x2F;propublica-is-h...</a><p>[0] <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=8505000" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=8505000</a> <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=10585009" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=10585009</a>

Can anyone shed light on why there has been a phenomenon of people finding SQL &#x27;too complex&#x27; and moving to noSQL? (Not sure if that&#x27;s entirely fair but, from the outside, it is what it looks like). Is it hype driven? Are courses at university not tending to cover SQL that much?

There just is no substitute for SQL. Some thoughts on what has given it a bad name:<p>1) The pervasive use of artificial keys. USE NATURAL KEYS. Unfortunately probably 99% of real-world databases were designed with artificial keys. I wish I could point to some literature on this topic. It is very rare and I only came to learn about this from a DBA who is well-versed in designing with natural keys. I&#x27;m trying to get him to publish more on this topic.<p>2) ORMs. This is just a bad practice. Their use in part derives from the awful schemas designed with artificial keys, requiring another layer of complexity to get a more intuitive model of the data. Fortunately for me over the past 3 years I&#x27;ve been doing almost all my application I&#x2F;O with F#&#x27;s SQL Type Provider, SqlClient, <a href="http:&#x2F;&#x2F;fsprojects.github.io&#x2F;FSharp.Data.SqlClient&#x2F;" rel="nofollow">http:&#x2F;&#x2F;fsprojects.github.io&#x2F;FSharp.Data.SqlClient&#x2F;</a> , which strongly types native query results, functions, and SPROCS. Just does not work if you need to construct dynamic SQL. I&#x27;ve been trying to goad the author into also providing meta data retrieval. That would be the icing on the cake.<p>3) SQL does not seem to be a required topic for undergrads. There are really no unsolved problems (of note), so it&#x27;s not interesting to academics.<p>4) Most app programmers don&#x27;t get much practice writing difficult queries or tuning problem queries, so that one time every 9 months when you do something hard, it is hard. (And again, often compounded by the complexity introduced by artificial keys.)

I&#x27;ve always thought the main reason NoSQL solutions became popular is that developers could finally get at and manipulate the data the way they wanted to.<p>I&#x27;ve known some very, very prickly DBA&#x27;s in my time who referred to the databases they looked after as &quot;My database&quot;. So they would say things like &quot;Don&#x27;t put junk in my database&quot;<p>And would give you endless grief over how you wrote your queries or asked you a million questions about why you needed a new table and why your proposed design was shit.<p>As a result, many of us devs tended to view SQL Databases as some sort of dark art. So in this regard, NoSQL is freeing at first glance.<p>But if I&#x27;m honest, once I got over my fear, the &quot;pros&quot; of NoSQL solutions in comparison to good ol&#x27; SQL seem to be relatively feeble.<p>I think it&#x27;s easier to get up and running with a NoSQL solution because there is far less friction when it comes to rapid prototyping of ideas, but things get complex pretty quickly.<p>I&#x27;d also say that for the vast majority of applications out there, the difference between the two will mostly be a wash.

I&#x27;ve been programming for most of my life. SQL has been a big part of my career. And I love it. It&#x27;s one of my top 5 languages.<p>It&#x27;s a nice, functional, declarative language in the vein of prolog and such. You just tell it the shape of the data you want, where to materialize it from, filter, aggregate, calculate the window of, etc... and the system figures out how to execute it as efficiently as it can. It beats out procedurally munging data by a long shot. It&#x27;s more concise for many operations than ML-like variants.<p>It&#x27;s a great tool to have. And understanding the underlying maths, relational algebra, is beautiful. I&#x27;ve found trying to implement your own rick-shod relational database is a good way to try to mechanically understand the theory. Then move on to implementing datalog... etc. The reason why SQL continues to stick around is that the fundamental theories are quite sharp! I&#x27;d appreciate a more concise syntax some days but overall I can&#x27;t say I&#x27;m displeased. It&#x27;s great!

One of my dreams is building a hybrid of RDBMS and Protocol Buffers. It would look like a bunch of nested structs that can be kept in memory, fetched from disk or over the network. The schema would be kept in .proto files, and you&#x27;d be able to reload it at runtime (existing data would be handled according to protobuf schema evolution, because each field in each struct is numbered and old numbers aren&#x27;t reused). Most joins would be gone because nested structs are allowed, but you&#x27;d still have an SQL-like language for more general queries (designing such a language when nesting is allowed is a bit tricky, but not very). Things like indices and transactions would also exist at every level (memory, disk, network) because they are useful at every level.<p>The end goal is eliminating impedance mismatch, because your current in-memory state, your RPCs and your big database would be described by the same schema language, somewhat strongly typed but still allowing for evolution. I have no idea if something like that already exists, though.

&gt; SQL and relational database management systems or RDBMS were invented simultaneously by Edgar F. Codd in the early 1970s.<p>Codd didn&#x27;t invent SQL. Donald Chamberlin and Raymond Boyce did.<p>&gt; SQL is originally based on relational algebra and tuple relational calculus<p>Maybe &quot;originally based on&quot;, but not &quot;an implementation of&quot;. For example, it is perfectly possible for an SQL query to return duplicate rows, which isn&#x27;t possible under relational algebra&#x2F;calculus, a relation being by definition a set of rows (or, more precisely, of tuples).

SQL is the second most underused tool we have with dealing with data. AWK being the first. SQL is great because the logic works with dealing with data and forces you to make good decisions earlier.<p>Everyone should spend a day learning SQL if for no other reason they the ability to think logically about data.

&quot;Why do we still use SQL&quot; and &quot;Why do we still use relational databases&quot; are two very different questions. They seem like much the same thing, because SQL is pretty much the only query language offered by relational database systems nowadays... so if you use SQL, you use an RDBMS, and vice versa. But other query languages used to exist. There was QUEL [1], for example. It seems to have fallen by the wayside; most people have probably never heard of it. I guess there is very little room for multiple languages in this particular space.<p>[1] <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;QUEL_query_languages" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;QUEL_query_languages</a>

&quot;It’s like how MailChimp has become synonymous with sending email newsletters. If you want to work with data you use RDBMS and SQL. In fact, there usually needs to be a good reason not to use them. Just like there needs to be a good reason not to use MailChimp for sending emails, or Stripe for taking card payments.&quot;<p>Wow, that&#x27;s a subtle, almost unnoticeable promotion of MailChinp. &#x2F;s

I came expecting a treatment on Structured Query Language, was disappointed when it turned out to be on Relational Database Management Systems. It doesn&#x27;t take a rocket scientist to figure out why RDBMSes won out over non-relational systems.<p>What I want to know is why nobody ever came up with a better query interface. Every abstraction I&#x27;ve ever seen was built on top of SQL.

Is it fair to compare the age of SQL to the age of JavaScript or that SQL has survive rapid change? SQL is a class of languages, while Javascipt is a specific language. The concepts behind JavaScript are older than SQL, and the modern SQL languages we use today are younger than 43yrs.

I&#x27;m not as old as some of you guys, been programming professionally for 11 years. The one constant in my life is SQL. I like it. My C# code is all obsolete now, the javascript code I wrote a year ago is obsolete, the Ruby code I&#x27;ve written never became a successful business. About half the PHP code I&#x27;ve written has been rewritten by now probably. But the SQL I&#x27;ve written is still living on, the tables I designed are still in use, the queries are still querying.

Just for fun: does anyone have any stories about initially using a SQL database for a project, later hitting problems, then switching to (or augmenting with) a NoSQL solution that solved those problems?

I&#x27;d really like for something like K&#x2F;Kx to pick up. For server applications, the dichotomy between DB and application seems so artificial for a lot of applications. Think Erlang + Mnesia, but with a fully relational model backed by language primitives.<p>I think LINQ with F#&#x27;s type providers would probably be what I have in mind (which works like JOOQ with a tighter integration).

The great thing about SQL databases is that the expected standard of performance is &quot;just works&quot;. Works all the time, for years, without trouble, even for the hard cases. All the major SQL databases, SQLite, MySQL, MariaDB, Postgres, Microsoft SQL Server, and Oracle, achieve that.<p>Contrast this with most webcrap. Or most of the NoSQL databases.

In my experience, (O)RDBMS and SQL are very good solutions for most of the business cases. I know a lot of projects that jumped on the &quot;NoSQL for everything&quot; train and eventually migrated partly to a RDMBS.<p>I often don&#x27;t understand why people try to avoid SQL by any costs instead of just learning and applying it properly. I don&#x27;t understand those &quot;we do SQL for everything&quot; teams either.<p>RDBMS in conjuction with NoSQL solutions can be a very powerful combination. We do a lot of Postgres + Redis + CouchDB in my projects.

A bit of a plug but I wrote about this a week or so ago describing SQL as the perfect interface. Databases change and evolve but since they all wrap the underlying engine in SQL it becomes very easy to use new technologies under the same interface: <a href="http:&#x2F;&#x2F;dangoldin.com&#x2F;2017&#x2F;04&#x2F;11&#x2F;sql-is-the-perfect-interface&#x2F;" rel="nofollow">http:&#x2F;&#x2F;dangoldin.com&#x2F;2017&#x2F;04&#x2F;11&#x2F;sql-is-the-perfect-interface...</a>

I think most developpers using SQL use it a bit like most office users using VBA (ie they know how to record VBA macros but not much more). Most developers know how to write basic queries so have a very superficial understanding but most likely know very little about the performance implications, structured indexes, nesting of queries, etc. Whereas I would expect developpers who claim to use C to have more than a superficial understanding of the language.

This remembers what my teacher said to me a decade ago: &quot;- Relational databases are the most successful software humanity have created.&quot;.

I love SQL and I believe it will have a big jump in popularity with microservices.<p>Each service should have a separate data source and being the sole responsable of a specific part of the data.<p>In this environment however a full RDBMS is a little an overkill.<p>The solution I am working on is RediSQL: <a href="https:&#x2F;&#x2F;github.com&#x2F;RedBeardLab&#x2F;rediSQL" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;RedBeardLab&#x2F;rediSQL</a><p>It is a Redis module that embed SQLite.<p>Redis is nowadays a common piece in any infrastructure.<p>The little module plugs into Redis and expose a new command REDISQL.EXEC that provide the ability to run SQL statement.<p>It is multithread, does not impact the performance of Redis, and very simple to use.<p>Great write performance, I got 50k inserts per second on my machine, that should be enough for most microservices.<p>I would love any kind of feedback on the module or if you need any help to get you started just open an issues.<p><a href="https:&#x2F;&#x2F;github.com&#x2F;RedBeardLab&#x2F;rediSQL" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;RedBeardLab&#x2F;rediSQL</a>

A lot of mentions to ORMs being the problem, them being poor abstractions, etc.<p>In my work with Django&#x27;s ORM I have run into problem queries as often as I have with doing SQL, and Django&#x27;s ORM has never let me down.<p>ORM keeps your thinking in line with your object oriented code, and I find it very easy and natural to use. Who cares if there are a bunch of artificial keys underneath, imo caching queries is a better solution to slow queries than trying to optimize SQL. So using the ORM is never a pain point.<p>The other advantage is automatic escaping to prevent SQL injection, which is still a top contender on OWASP&#x27;s list. I never have to worry ahout SQL injection when using the ORM.<p>Maybe other ORMs are poor solutions, but at least with Django I have been very happy using it.

My favorite SQL fact: The San Carlos Airport (which is about 1mi from Oracle HQ; a plane losing power on takeoff would very plausibly crash into the towers, and in fact one did fall into the Redwood Shores lagoon a few years back, likely a choice by the pilot to avoid hitting a populated area) is KSQL, so airport code SQL. And its existence predates Oracle&#x27;s headquarters being located there. It&#x27;s just a coincidence.<p>KSQL and Oracle towers: <a href="http:&#x2F;&#x2F;www.bayareapilot.com&#x2F;IMG_0317%20Large%20Web%20viewnearingSQL.jpg" rel="nofollow">http:&#x2F;&#x2F;www.bayareapilot.com&#x2F;IMG_0317%20Large%20Web%20viewnea...</a>

SQL is good, but it shows its age. Today somebody should come up with something statically typed and more functional (meaning using lambda calculus as a starting point).<p>The biggest pain points of SQL (IMHO) are:<p>- lack of statically typing guarantees (for example, no guarantee that a certain table has certain column)<p>- bad capability to abstract over parts of the data model (for example, queries have to specify the table that they query)<p>Both of these can be resolved with use of good enough functional language. There are projects like that in the FP&#x2F;Haskell community (e.g. Ermine), but it&#x27;s fragmented.

Good old SQL. Your reliable friend that always shows up with the right amount of booze and gas money, and which, when you stop to think about, basically hasn&#x27;t ever majorly fucked up around you.

I think back to my first programming job and some of the stuff I used then: Perl, early versions of PHP and other such tools that I haven&#x27;t used in a long time.<p>Two of the things I still use to this day, though, are:<p>* Postgres<p>* Emacs

While SQL has a large class of uses, and also a smaller class of problems for which better solutions exist (CAP constraints, time series and other massive self-join apps, session storage ...) the reason we&#x27;re going to use SQL in another 40 years still is that there&#x27;s no cross-vendor standardization effort going on anymore (NoSQL vendors don&#x27;t seem to find it necessary to drive sales and market growth, and customers don&#x27;t demand it either).

Is there any work on a successor to SQL (just the language, maybe as an optional frontend)? I am not a fan of SQL, it works great for simple queries, but fails (in my opinion) for more complicated ones. They get way to complicated and hard to understand for something that would be easy in other languages. This is not a critique of relational databases, only the language.

The amount of confusion in the comments highlights why we don&#x27;t have anything better.<p>SQL is just a language, it specifically stands for Structured Query Langauge. It has <i>nothing</i> to do with the underlying database.<p>Relational databases all implement SQL because that&#x27;s what the language was originally created for but it&#x27;s just an interface. Relational databases can also implement other interfaces like mysql with its X protocol. Other database types like key&#x2F;value, document, graph, columnar, time-series, RDFs, etc can also implement SQL and many are starting to for easier interoperability, like Cassandra with CQL.<p>There is definitely potential for a better query language and there are examples like ReQL and GraphQL but SQL is still just fine for most use-cases.

Most(?) of the articles I read on HN, and then their comments, always seem to put NoSQL in a bad light when people use it for things that &quot;should&quot; be in SQL<p>What _are_ the &quot;correct&quot; use cases for NoSQL? Everything has always been relational data for me

Does anyone know how to generate a visualization like the one under reason #2 (&quot;Battle Tested&quot;)?<p>Here&#x27;s the gif: <a href="http:&#x2F;&#x2F;imgur.com&#x2F;K5a7U9O.gif" rel="nofollow">http:&#x2F;&#x2F;imgur.com&#x2F;K5a7U9O.gif</a>

Unlike with imperative languages which are dozen a dime, there&#x27;s almost no alternative to SQL when it comes to relational languages.<p>That being the case, people rarely even think about whether SQL is a good language or a bad language, whether it&#x27;s lacking something etc.<p>But once you actually try thinking about it, it turns out that it&#x27;s a pretty well designed language and any alternatives you can think of are usually much inferior to it.

I think it&#x27;s funny when the author compares SQL being most loved with least dreaded. They&#x27;re the same thing. The percentages all add up to 100.

SQL&#x27;s just so much more flexible than the competing query languages, there&#x27;s not a true alternative to it. One syntax does a decent job for transactional processing, key&#x2F;value tables, heavily relational data, massive analytical processing, data warehousing, etc. It&#x27;s pretty ugly, but it&#x27;s flexible enough that you can get the job done even if you need to do something complex.

Since a lot of people are thinking about this: Is there a good way to compose SQL?<p>I see a bunch of repeated items in many SQL queries, things that would be functions in another language.<p>One of my colleagues pointed out that this is indicative of poorly formed queries. What do you all think?

When we approach a customer with our database, SQL compliance is super important to them.<p>Some of our competitors used to be &#x27;SQL-like&#x27;, and even they swapped to using full SQL.<p>I think the fact that SQL is based on solid mathematical principles really helps it stay relevant.

Should mention SQL wasn&#x27;t the original RM language: <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Alpha_(programming_language)" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Alpha_(programming_language)</a><p>;-)

Possibly relevant: Modern SQL is a good resource, by Markus Winand, on the newer aspects of the language. <a href="http:&#x2F;&#x2F;modern-sql.com&#x2F;" rel="nofollow">http:&#x2F;&#x2F;modern-sql.com&#x2F;</a>

You can get away with not using SQL if using the functionality from LINQ or Java streams but as a DBA I feel most at home with SQL.

SQL was my entry point into software development, and I have a somewhat emotional attachment to it. And I&#x27;m quite glad that it worked that way.<p>SQL, relational theory, and set theory are a great place to start understanding how to work with data. And a great way to start understanding software.<p>All software deals with data. If you don&#x27;t have a good understanding of data, you are never going to have a good understanding of software.<p>One of the best books I&#x27;ve ever read was Applied Mathematics for Database Professionals by Lex de Haan, and Toon Koppelaars. I think that&#x27;s the database equivalent of SICP. You need to read it and understand it if you want to seriously deal with data. And you want to if you want to write software.<p>I&#x27;m obviously biased because of the way I got into things, but I look at things as a top-down vs a bottom-up point of view. I was a violinist and music theorist before I got into technology, and the bottom-up approach has always resonated with me.<p>In classical music, pretty much everything bubbles up from a baseline foundation and a structure; the stuff at the top that you actually see is a result of that structure. You don&#x27;t start with some notes that you want to play on an instrument and then go and try to find a structure that supports those notes. You go bottom-up. You lay the foundation and build on that.<p>It was easy for me to map that idea of musical theory onto a database early in my career. And I moved up in the stack as I needed to. I started by building things entirely in SQL. You want complex statistical analysis? Sure, I&#x27;ll do that . . . in SQL. Because I didn&#x27;t know any better.<p>Then I found out that there are actually other languages that can do certain types of things much better. R, Python, C#, etc.<p>11 years later, I&#x27;m now very capable in a number of languages, and I don&#x27;t suck. Along the way, I&#x27;ve had to put a lot of effort into learning the things I would have got from a comp sci degree program, and I&#x27;m probably not the best at certain types of software challenges.<p>I use noSQL stuff for caches and data warehouses, I use some of that for offloading traffic and keeping the reads separate from the writes. But there isn&#x27;t a project that I touch that doesn&#x27;t involve SQL in some way.<p>SQL is incredibly useful every day. Learning it, knowing it, understanding it, is a bare minimum for people I hire.<p>If you have a comp sci degree, and you don&#x27;t know SQL, I&#x27;m going to probably write you off. If you have a liberal arts degree of some kind, and you do know SQL, I&#x27;m probably going to hire. You can learn everything else on the job.<p>None of that is an excuse for the totally shitty article linked here.<p>We use SQL today because it&#x27;s good and it works. Not many languages can say that these days.

SQL as language is one thing. The fact that RDBMS systems with some consistence and isolation guarantees is a different story.

Meh. I&#x27;m not one to sing the praises of SQL so highly. I understand it&#x27;s history and its use - but it&#x27;s also really, really difficult to understand and figure out complex SQL queries.<p>Personally, I thought REQL was a really interesting take on query languages. As a developer, it allowed you write much cleaner code. You barely need an ORM. REQL kinda sucks for analysts at first, but in the long run, it makes writing complicated queries much, much easier.

It lists Redis in there for the SQL ones. Redis has SQL now?

I&#x27;m wondering what is the revenue of SQLizer?

i guess a lot of performance was sacrified and a lot of optimizations made, witch make sql very fast in the current era.

When the best SQL guy is a dude that memorized a bunch of language hacks to get the underlying algorithm to be more efficient I question the design principles of the language. Instead wouldn&#x27;t it be better for the language to explicitly allow the user to apply algorithms or procedures to make things more efficient rather than apply hacks?<p>The language is too heavy of an abstraction away from what&#x27;s really going on under the hood. In a way it suffers from the same issue as functional programming. Not saying functional programming&#x2F;sql is bad but... it has issues like almost everything.

It works.

SQL is really good at projecting &amp; selecting simple data. My rule of thumb has been to use it first for any per projects and but as soon I need more than one table, think deeper about the data and switch to NoSQL if I need to represent complex data structures or have document storage needs.<p>Its still amazing how far you can go with a single table and few tweaks to a postgres instance.