TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

Why I love databases

238 pointsby strzalekover 10 years ago

19 comments

jandrewrogersover 10 years ago
Nice article. I love databases too for similar reasons but, as someone that designs database engines, some of the technical points are off the mark. I never really stop learning in this area, the technical range is incredibly deep and nuanced.<p>Some of the points that caught my eye as being quite off:<p>- Contrary to footnote 2, modern database designs bypass the OS file system cache and schedule their own I&#x2F;O. This has an enormous performance impact versus the OS cache (2-3x is pretty typical) and is a good litmus test for the technical sophistication of the database implementation. It is the primary reason many open source database engines, even &quot;write-oriented&quot; ones, have relatively poor write performance to disk.<p>- The three data retrieval models enumerated are textbook but if you were designing a new engine today you probably would not use any of them for a general purpose design. Modern spatial access methods (ex: Hyperdex, SpaceCurve, Laminar) are superior in almost every way though the literature is much sparser. Also, a number of real-time analytical databases use bitmap-structured databases (ex: ParStream), which are incredibly fast for some types of query workloads.<p>- Many distributed database challenges are a side effect of &quot;one server, one shard&quot; type models. It is not necessary to do things this way, it is just simpler to implement; some distributed database systems have thousands of shards per server. The latter model is operationally more robust and better behaved under failure and load skew.<p>- Tombstones are usually trivial if the database engine is properly designed. Complications are a side effect of poor architecture. The big challenge for tombstones is deciding when and how tombstoned records are garbage collected. It is outside the scope of the normal execution pathways but you also don&#x27;t want a Java-like GC thread in the background.<p>Of course, any of these is a long blog post in itself. :-)
评论 #8580623 未加载
评论 #8581155 未加载
评论 #8581154 未加载
pavlovover 10 years ago
I hate databases. People tend to have way too much faith in them (or their surrounding marketing), and thus make poor database choices that don&#x27;t actually fit the shape of their data. Persistence is fundamentally the programmer&#x27;s responsibility; a magic box behind a socket can&#x27;t design it for you.<p>Most applications I&#x27;ve seen wouldn&#x27;t even need a database, but apparently a lot of programmers are conditioned into believing that writing anything to disk must involve building a database query string and transmitting it over a socket to another process which parses the string, executes it on an interpreter and stuffs the extracted data into a generic 1970s data model that finally gets written to disk in an opaque format from where it can only be retrieved by sending more strings over sockets. This stuff made sense when 1MB was a huge amount of memory, but today it&#x27;s just not necessary.
评论 #8581037 未加载
评论 #8581547 未加载
评论 #8582024 未加载
评论 #8581537 未加载
评论 #8581723 未加载
jeeyoungkover 10 years ago
Thank you everybody for your valuable comment. Like I&#x27;ve said in the post, this was my first attempt to explain my passion. I read every one of your comments, and spent the next few days fully understanding their implications.<p>This post is a bit biased towards what I&#x27;ve interacted with. I think I do have a good sample of various workflow, but there are obviously large number of databases and use cases that I&#x27;m missing, and my view of what is &quot;modern&quot; may not incorporate a lot of bleeding edge theory and technologies.<p>Operational complexity is definitely my #1 concern. The service I originally maintained is a OLTP system, located at the top of the service &amp; data dependency graph. Availability is the top concern.<p>The current system I&#x27;m writing is metrics database. The operational burden is much lighter. It is almost a leaf node in the service &amp; data dependency graph, and I can take downtime to restart the cluster. A very different workflow, indeed.<p>Thanks!
gfodorover 10 years ago
&quot;Designing Data-Intensive Applications&quot; is shaping up to be an excellent treatement of modern databases and their underpinnings. It&#x27;s at an excellent level of abstraction, deep enough to convey database internals while high level enough (so far at least) to be able to cover a wide variety of database systems. It also has its feet firmly planted in database history, and is NoSQL-koolaid free. Highly recommended.<p><a href="http://shop.oreilly.com/product/0636920032175.do" rel="nofollow">http:&#x2F;&#x2F;shop.oreilly.com&#x2F;product&#x2F;0636920032175.do</a>
评论 #8580757 未加载
Slackwiseover 10 years ago
I&#x27;ve always loved databases, but after having discovered write-only timestamped databases like Datomic, I can&#x27;t imagine going backwards. It&#x27;s a real shame that Datomic isn&#x27;t fully open source.<p>(Aren&#x27;t BigTable and Spanner also write-only and timestamped?)
评论 #8581234 未加载
评论 #8580775 未加载
评论 #8581243 未加载
quonnover 10 years ago
&quot;The study of databases intersects almost every topic in computer science&quot; - I&#x27;ve heard this before, especially for Compilers. But it has been false for a long time, CS is far more diverse now. For example, how do Databases intersect AI&#x2F;Machine Learning&#x2F;Computer Vision, Computer Graphics, Numerics&#x2F;Simulation, Robotics, Bioinformatics, Computer Architecture or Cryptography?
评论 #8580470 未加载
评论 #8581460 未加载
评论 #8581580 未加载
评论 #8580808 未加载
评论 #8580463 未加载
Pxtlover 10 years ago
I love databases, but I loathe SQL. And no, I don&#x27;t mean NoSQL is better - that&#x27;s throwing out the baby with the bathwater.<p>To me, SQL is the Common Lisp of relational languages - a brilliant invention of its time that has since long-overstayed its welcome and should be replaced by modern considerations of the problem it solves. The difference is that there are a million rethinks and descendents and redesigns of LISP out there that happily threw out the mistakes and made great strides in the language. You could argue that <i>every</i> modern programming language is a descendant of Lisp thanks to the prevalence of great concepts like lexical closures. SQL, on the other hand, has a teeny tiny few spiritual fringe descendents like the various attempts at Date and Darwin&#x27;s &quot;Tutorial D&quot;.<p>I love the relational model, but who says the only way to manage the relational model is this hoary old thing? It&#x27;s immensely frustrating that every implementation of SQL bolts on a tacky and half-assed procedural language, but doesn&#x27;t solve simple underlying frustrations.<p>Simply accessing related objects is immensely wordy for a &quot;relational&quot; language. In an algol-derived language, I can say Group.Manager.Person.Address.PostalCode to walk the graph. In SQL, I have to deal with a zillion joins.<p>Yes, some SQL variants let you join by the foreign key name to make the join a little more terse, but it&#x27;s still hairy compared to every modern functional or procedural language.<p>And the APIs - maybe the reason so many sites have SQL injection problems is the hideous APIs. Ever tried to build a WHERE IN (id1, id2, id3... idN) statement with a proper parametrized queries? Holy crap what a icky mass of boiler plate. I mean, it&#x27;s not a <i>hard</i> problem, but how many times have you solved it, and how many times have you found a tedious bug in your solution? Just give me a proper way to concatenate the parametrization <i>inline</i> with the query FFS.<p><pre><code> db.RunQuery(&quot;&quot;&quot; SELECT * FROM MYTABLE WHERE ID IN + &quot; + db.SomeParameterListFunc(a, b, c) + &quot; ORDER BY HOLYCRAP_WAS_THAT_SO_HARD&quot; &quot;&quot;&quot;); </code></pre> The above syntax would be trivial in any language with operator overloading on the &quot;+&quot; sign, on the off chance that your SQL dialect is so messy it&#x27;s impossible to safely build a properly-escaped initializer for the list containing a,b and c in text form.<p>And that&#x27;s not even getting into real actual first-class language support like ORMs give you.<p>And speaking of APIs, the fact that a single &quot;SELECT&quot; is the baseline operation... that you work on one resultset at a time. I don&#x27;t want a single pile of rows. This is not an excel spreadsheet, it&#x27;s a relational database, and that means I want a <i>graph</i> of data. I don&#x27;t want to write three queries to get my Customers, their Personnel, and their Addresses, nor do I want a single row of CustomerPersonnelAdddresses. Once and Only Once is good for the data, why the heck isn&#x27;t it good for result sets?<p>Where&#x27;s the code reuse? Why can&#x27;t I have a pile of SELECTs and a pile of WHEREs and combine them however I see fit? Oh right, I can use a VIEW... but see the previous point, a VIEW is a single glorified Excel spreadsheet, not a proper graph of data. If I want to bundle a bunch of SELECTs together, I have to just write a stored procedure, but then I can&#x27;t use the proc with a JOIN statement against my VIEW that provides a custom WHERE clause. You could do something monstrous with table-valued parameters, I guess, but those aren&#x27;t generally well-supported at the API level. This is not a hard problem in every modern language (except Go, of course - yes, you <i>do</i> freaking need map&#x2F;reduce&#x2F;filter).<p>Namespaces. Real, actual, organizational tools for your giant list of 9000 tables and their related objects. No, schemas don&#x27;t freaking count - you can&#x27;t nest them and they&#x27;re overly tied to the security model - using schemas for organization instead of security leads to madness, besides the fact that you can&#x27;t nest them.<p>And of course, so many common problems simply <i>aren&#x27;t</i> nice to work with using the relational model. How do I make a nice audited row where I have the full history of all the row&#x27;s changes? Well, I can insert it every time, but that&#x27;s a lot of wasted space. Yes, again, there are ways to do this, but it&#x27;s something I&#x27;d expect to come out-of-the-box since it&#x27;s such a common problem. Common problems should be solved by the standard library. But SQL can&#x27;t solve things like this with the standard library, because a SQL standard libraries are limited to crude things like functions and views and procs and not actual large-scale reusable constructs. It&#x27;s like a programming language where they gave you a bunch of general tools for manipulating unicode points and dynamically sized arrays but no coherent &quot;string&quot; object.<p>Or <i>trees</i>. Holy crap, you have a &quot;relational&quot; database where a relationship like a &quot;tree&quot; is a nightmare to actually query out! I know that&#x27;s not what &quot;relational&quot; means, but still - this ins&#x27;t exactly a rare edge-case, y&#x27;know? But it&#x27;s not in the standard library because the standard library is limited to crude objects like data-types, functions, procedures, etc. that work below the row-level. Any concept of reusable <i>schema</i> concepts is completely left off the table.<p>&#x2F;rant
评论 #8582013 未加载
评论 #8582857 未加载
评论 #8583045 未加载
评论 #8581949 未加载
评论 #8594082 未加载
readover 10 years ago
&gt; You cannot give up partition tolerance.<p>It&#x27;d be more accurate to say you don&#x27;t want to give it up all the time. You don&#x27;t want CAP; you want PACELC.<p><a href="http://dbmsmusings.blogspot.com/2010/04/problems-with-cap-and-yahoos-little.html" rel="nofollow">http:&#x2F;&#x2F;dbmsmusings.blogspot.com&#x2F;2010&#x2F;04&#x2F;problems-with-cap-an...</a>
评论 #8582283 未加载
rattrayover 10 years ago
I wonder why RethinkDB hasn&#x27;t gotten a mention here. Have folks here used it? Thoughts?
cfolgarover 10 years ago
Since we&#x27;re discussing databases...is there any &quot;golden standard&quot; learning resource&#x2F;introduction to PostegreSQL? As a college student, I do not have much experience with them yet but I am aware of how important it would be to be comfortable with them in your day to development. Something tells me that my usual approach of diving in and tinkering by building out an idea wouldn&#x27;t serve me as well for db&#x27;s; it just seems to me that there&#x27;s some fundamental database concepts that I would be missing if I went down that path.<p>Any advice as to where to start a structured approach to learning about databases would be highly appreciated :)
评论 #8581324 未加载
评论 #8592639 未加载
Roboprogover 10 years ago
I have lived a very sheltered existence. I have never worked on an application which had a database cluster or sharding, rather than just running on a single server.<p>Of course, the servers are a little bigger now than they were back around say 1990.
评论 #8581313 未加载
foglemanover 10 years ago
I love databases for the data they contain. And for the ability to make sense of that data more easily when it&#x27;s in a nice, structured format.<p>I don&#x27;t care as much for the dev-ops side of it.
pm90over 10 years ago
What I find both fascinating and scary about databases is how to choose between the wide variety of databases without understanding exactly how they work? And it doesn&#x27;t help that there are new databases springing up all the time.<p>Is there a way for application developers to understand these databases quickly without spending weeks working with them?
评论 #8580909 未加载
评论 #8581228 未加载
评论 #8581044 未加载
tkyjonathanover 10 years ago
I like SQL. It makes sense to me, personally. Even when I get really unwieldy, I can always take a step back and break it down to parts. If I&#x27;m allowed to use temp tables, I can do almost anything in it and usually a lot faster than developers in their language can.
metaphormover 10 years ago
this was a great article. extremely helpful to me, coming from a position of being a server-side application developer often tasked with getting several disparate data-stores talking to each other.<p>the author does a great job of showing the full depth of the field while providing useful hooks and links for further study. this one made my bookmarks folder. i&#x27;ll surely be going back to check it out again.
smartpantsover 10 years ago
This is what i needed. Great read
digital-rubberover 10 years ago
In my opinion, something more important, does the database love your data? Only then you can truly love your database.
marknadalover 10 years ago
I&#x27;m glad he loves databases, databases have been the bane of my existence.<p>However, the torment they have given me has also lead to a similar fascination - and now I&#x27;m writing my own database! So I&#x27;ve become very familiar with the topics he writes on, and they are very good points for anybody interested in the subject.<p>Why would I write my own database? Because databases are hard, and I am determined to make them easy (even if that means me sacrificing years of my life into doing all this crazy research). Check out <a href="http://github.com/amark/gun" rel="nofollow">http:&#x2F;&#x2F;github.com&#x2F;amark&#x2F;gun</a> !<p>- CAP Theorem, he is correct, P cannot be sacrificed. GUN is AP with eventual consistency. The beauty of this, though, is that you can always build strong consistency out of eventual consistency (it just requires knowing X amount of peers in advance, and doing a trivial lock until you&#x27;ve heard back from all of them - in fact, I do this in one of the example apps) but you can never go from strong consistency down to an eventually consistent system.<p>- Distributed Systems, this is incredibly incredibly important. I cannot repeat this enough times, there should be no &quot;master&quot; or &quot;single source of truth&quot; in any database. If there is, you&#x27;re going to have a nightmare of a time being woken up at 3am to fix it when it crashes (my personal experience with other databases). Why? Because single points of failures will fail, centralized systems suck. Solution: Distribute and Decentralize! We make this easy for you.<p>- Correctness vs Efficiency, as he says, Paxos is difficult - all of them, Raft, Quorum, leader election, etc. DO NOT USE THEM unless you are Google, Amazon, Walmart, or what not. Even then, do not use them. Instead, I&#x27;ve solved this challenging problem by developing a new Conflict Resolution system that (very poorly) can be summarized as Vector Clocks + Timestamps, you get the advantages of both without either of their weaknesses. What his means is that data integrity is guaranteed because every machine is using a deterministic algorithm, without any extra gossip between machines. Let me repeat, you&#x27;ll get the same result on every machine, eventually consistent, without any multi-machine coordination. This means every peer is a master, and that is awesome, even if you are running it on an ephemeral server&#x2F;cloud - completely resilient to failures, terminations, restarts, and reboots.<p>- Empowering the App. Yes. Databases should serve you, not the other way around. Answers to his questions about abstractions are at <a href="http://github.com/amark/gun" rel="nofollow">http:&#x2F;&#x2F;github.com&#x2F;amark&#x2F;gun</a> .<p>- Operational Challenges. This is where I diverge from him. If something seems wrong, like things suddenly becoming slow, you can easily just restart it without any damage&#x2F;harm&#x2F;failure occurring. And then you can look through your logs, taking your time, to see what went wrong.<p>- Basic Building Blocks. Because GUN is a graph database, you get key-value like access, as well as documents and relational styles. That is because mathematically graphs are the superset of relational algebra and hierarchy trees.<p>Happy to answer any questions!
评论 #8581528 未加载
评论 #8582754 未加载
评论 #8581815 未加载
评论 #8581608 未加载
评论 #8581288 未加载
评论 #8583107 未加载
评论 #8582009 未加载
MrBraover 10 years ago
Well, intro was promising and full of energy. Like &quot;I am really going to transmit you some good love for databases and explain why it&#x27;s so stimulating to deal with them!&quot;, but then right after this sparkling start it&#x27;s all just about same old redundancy, consistency, scaling...