TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

Truth First, or Why You Should Mostly Implement Database-First Designs

205 pointsby MarkusWinandalmost 7 years ago

21 comments

ben509almost 7 years ago
This is a really important piece for developers to read.<p>We get very used to the ability to blow away everything and start from scratch, and that&#x27;s very often the right answer for build tools, but the database is fundamentally different from that, and you <i>need</i> to internalize this.<p>A database, in most cases, is the business. It&#x27;s essentially a collection of all the contracts made with customers, and your mode of thinking &quot;I can blow this away and start over&quot; that you&#x27;ve learned from all sorts of build tools is completely contrary to that.<p>An infamous example of this is in Active Record migrations[1], and possibly some other migration tools.<p>&gt; Note that we define the change that we want to happen moving forward in time. Before this migration is run, there will be no table. After, the table will exist. Active Record knows how to reverse this migration as well: if we roll this migration back, it will remove the table.<p>These are smart people, and indeed there are some <i>rare</i> cases where this is the right thing to do. But only our strongly ingrained belief that data is ephemeral would cause us to think this should be the normal behavior, that we&#x27;d want to blow away customer data to force the database to track what version X of the code thinks it should.<p>[1] <a href="http:&#x2F;&#x2F;edgeguides.rubyonrails.org&#x2F;active_record_migrations.html" rel="nofollow">http:&#x2F;&#x2F;edgeguides.rubyonrails.org&#x2F;active_record_migrations.h...</a>
评论 #17251218 未加载
评论 #17249623 未加载
评论 #17253084 未加载
评论 #17250996 未加载
评论 #17249173 未加载
Diggseyalmost 7 years ago
No, you shouldn&#x27;t use a code first, <i>or</i> a database first design.<p>The article explains the problems with the code-first approach very well, but there are just as many problems with the database-first approach:<p>- There is never just one database. There will be your production database, a staging database, one or more testing databases, local development databases, etc. This makes the database useless as a single source of truth.<p>- When you write code against your development database, there&#x27;s no guarantee that it will be the same as the production database, and you shouldn&#x27;t be able to connect to your production database while developing locally to verify that. Usually you want to deploy identical build artefacts to staging and production, so there&#x27;s a danger that your staging database gets out of sync with production (eg. after an issue was caught and improperly rolled back on staging) and then you deploy your fully tested code from staging to production and your entire system breaks down.<p>- When you need to restore a backup of your database, you need to somehow find a compatible version of the code to deploy as well.<p>My recommendation is to use your migration scripts as your source of truth. Each service should access separate schemas within the database, or at the very least separate tables. If you really need multiple services to share tables then those services should be versioned and deployed as one.<p>The database should have a table which stores a list of the migrations which have been applied, and you should use a migration tool such that when you deploy, outstanding migrations are applied as required.<p>If you want something easier to read than the history of migration scripts, then you should keep an up-to-date schema <i>as well</i> next to your code, and verify as part of CI that running all the migrations results in that exact schema.
评论 #17252946 未加载
评论 #17252229 未加载
nikisweetingalmost 7 years ago
From a startup perspective I strongly disagree, I think the UI should almost always be designed first. The HTML&#x2F;CSS&#x2F;JS frontend that the users see comes first, and have it hooked up to an empty backend that does nothing. Then once the UI requirements are fleshed out and the final specs nailed down, only then the database schema should be designed. The backend glue e.g. Django&#x2F;Rails comes last.<p>I find this approach has led to the fewest number of code rewrites and database migrations needed to get from initial mockup to finished product, because UI requirements can change much faster and are more fickle than DB schemas and backend code.
评论 #17253311 未加载
评论 #17252326 未加载
评论 #17258907 未加载
评论 #17252685 未加载
WorldMakeralmost 7 years ago
My two cents: always source control your databases.<p>I don&#x27;t care if you are code first or database first, but I do think database schema and migration scripts are code artifacts and need to be source controlled <i>somewhere</i>.<p>You should be able to track who made a change and when and what for, and maintaining by hand a &quot;commit list&quot; in a comment at the top of a stored proc doesn&#x27;t count.<p>You shouldn&#x27;t need a full backup of a Production database to spin up a new test server, and you shouldn&#x27;t need to rifle through every developer&#x27;s documents and desktop folders looking for random .sql script files to figure out if you are missing a migration&#x2F;update&#x2F;refactor script somewhere.<p>(The argument that you rarely spin up a new Production server is I think tempered by how many developers and testing environments you need. Setup time for developer&#x2F;testing instances indicates a part of how long onboarding new developers may take; and here I lean to the side of developer productivity and good backups.)<p>Most code first methods give you source control &quot;for free&quot;, which makes them easy to recommend to junior developers &#x2F; small startup projects.<p>Visual Studio has decent SQL Server Project support to help with the database-first crowd, as one option I&#x27;ve relied on heavily. Unfortunately, I&#x27;ve yet to see an open source, more database agnostic relative to it.
ams6110almost 7 years ago
First supervisor I had in my 2nd job out of school had a mantra: get the data model right and everything else falls into place.<p>Has worked for me for 30+ years.
评论 #17248250 未加载
评论 #17249299 未加载
评论 #17250909 未加载
sixdimensionalalmost 7 years ago
One use case for when you may consider code-first - pure experimentation and fast iteration around the data model that supports your application.<p>I have used both models, and did database-first extensively, for at least 15 years. However, having been working in .NET lately and having EF Code First migrations (painful, but read on), I did discover a sort of hybrid model that worked.<p>Eventually, I believe you will always end up at database-first, even if you start code-first, unless your database is relatively small (by # of table&#x2F;procs&#x2F;views count) or limited in scope. But as applications grow, multiple databases become a reality, the database might grow truly huge(# of tables&#x2F;procs&#x2F;views), you might need customization to your database platform, etc. - limiting it by the application &#x2F; code-first becomes impossible.<p>Where code-first works well, for me, is limited scope OR, more importantly, when you need really fast iteration for prototyping and don&#x27;t want to think about building the tables. I&#x27;ve found that use case, code-first can be really helpful to think in code, let the system generate the database, at least enough to get to an MVP or PoC type-level of work product. Then, once it&#x27;s working enough, to really take it all the way, switching to a database-first approach to do all the customizations to a particular database platform&#x27;s strengths, etc. can work - I&#x27;ve done it myself on several projects successfully.<p>The tooling is almost there to do this... just start code first, and then once you have your system bootstrapped, generate the database first model and switch. But, you have to be careful with naming things so that names of things in your code match up (or can easily be updated) if you switch from code-first to database-first - something that not all code generators let you control completely (most template-driven ones do, though).
steve-chavezalmost 7 years ago
If you&#x27;re using PostgreSQL, I invite you to try PostgREST <a href="https:&#x2F;&#x2F;postgrest.org&#x2F;en&#x2F;v5.0&#x2F;intro.html" rel="nofollow">https:&#x2F;&#x2F;postgrest.org&#x2F;en&#x2F;v5.0&#x2F;intro.html</a> which takes the database-first approach even further, with PostgREST you develop all your business logic in SQL, there&#x27;s no code generation, no ORM, which no matter how elegant will always impose a performance and abstraction overhead(to do an aggregate you have to read the ORM docs, learn&#x2F;understand the abstractions, add the code, all of this in addition to what you already know how to do in SQL).<p>Check the API <a href="https:&#x2F;&#x2F;postgrest.org&#x2F;en&#x2F;v5.0&#x2F;api.html" rel="nofollow">https:&#x2F;&#x2F;postgrest.org&#x2F;en&#x2F;v5.0&#x2F;api.html</a> for more details on all the features that work out of the box, no ad-hoc code needed.
mnm1almost 7 years ago
I couldn&#x27;t agree more. The code first design is what kept me away from Rails. I&#x27;m sure they have sensible approaches as well, but unfortunately I couldn&#x27;t get over my prejudice that anyone who teaches the creation of the database through the app&#x27;s code instead of actually designing the database is someone I don&#x27;t want to follow or learn from and whose framework I don&#x27;t want to use even if it provides alternatives (I hope it does). Database design in a CRUD app is by far the single most important task. Show me your code and I know nothing, show me your data structures and I understand the whole app never applied more.<p>The consequences of not following this advice I see everyday at my current job where we are stuck with an ORM that needs at least 3-4 files modified for each entity change and that requires management using an external GUI tool. The ORM adds 10x overhead to already unoptimized SQL queries so optimizing the queries is never a problem because the system&#x27;s overhead is so slow even if it was possible to make the queries run in zero time, the overhead would still kill us. 10x. Query is 10ms and overhead is 100ms. And the trade-off for this? Nothing but a shitty ORM that makes no sense (Doctrine 2&#x2F;PHP) and slows down development a few times over as well. Changing even the name of a column requires editing multiple files. We have camelCase, snake_case, and kebab-case for all our entities and depending on where you are in the system, a different case is required. Using the ORM to generate entities after you&#x27;re in production? Don&#x27;t make me laugh. That&#x27;s impossible. And this isn&#x27;t even the worst ORM I&#x27;ve used (though it&#x27;s close). Everything the author mentions and then a whole bunch of stuff that&#x27;s a million times worse has resulted from this drive to abstract away the most important component, the database, and not have it be the single source of truth. Frankly, it&#x27;s fucking stupid and beyond maddening, not to mention slow and due for a rewrite. Never again will I tolerate this kid of design that ORMs encourage or the use of ORMs at all, especially if they replace solid database design. I&#x27;ll leave that kind of fuckery and stupidity to the rookies.
评论 #17250975 未加载
评论 #17255685 未加载
评论 #17251623 未加载
vinceguidryalmost 7 years ago
What I want is a doctrine that gives an ideal approach for data management, similar to how 12 factor gives an ideal approach for overall application architecture management. Data architecture, even though it&#x27;s a subfield of application architecture, is important and vulnerable enough to need it&#x27;s own doctrine.<p>What I like about 12 factor is that you don&#x27;t have to get it all done at once. The concepts start high level and you can dive into the details as they become necessary. Initially, you can get by with dotenv for configuration management, and slowly build out more tooling <i>as you need it</i>.<p>But data management seems to need a new lexicon and set of organizing principles. How do I think about the varied aspects of database design in a way that I can be reasonably sure won&#x27;t bite me in the ass before I can get around to scaling it? How do I pick a database, what are the relevant decision criteria? At what point do you want to introduce different kinds of databases, like say a graph database or ElasticSearch?
评论 #17253385 未加载
soulnothingalmost 7 years ago
Jooq has become my favorite ORM with how bare metal it is. I have it set up in gitlab pipelines to auto build my schema into an ORM. Just kick build on a sql change, and then inherit the new package.<p>I saw an ancillary remark about prod connections, being bad. I generally agree. I usually build mine off of my staging environment. Assuming a staging database schema, was promoted to production. Then I have a sister build job, that kicks off my jooq build, trigger on the promotion task.<p>The biggest thing is the professional license for several databases. But it&#x27;s not that bad really.<p>It also goes to one of my thoughts as well. The data model is crucial to the application. This is like the foundation for your home. I always try and hammer in talk about the data model first. Where did it come from, where next, and finally where do you want it to go.
评论 #17251670 未加载
srikualmost 7 years ago
Sometimes, I get the feeling that we&#x27;re all in the story of the &quot;blind ones and the matter of the elephant&quot;. Design UI first? Design data model first? Design code first? Write documentation first? Write tests first?<p>How many things can we actually do &quot;first&quot;? All of them are important. Writing documentation tells you about what you might need implemented. So does designing UI, but that might still leave out what users need. Which means we probably don&#x27;t know enough about the data model anyway. Or the code.<p>So ... here is one more blind person&#x27;s view - iterate as quickly as you can on everything.
TimJYoungalmost 7 years ago
I&#x27;m not sure that the statement &quot;All generated DDL is only an approximation of that.&quot; is necessarily correct and is database engine&#x2F;server-dependent.<p>It&#x27;s typically pretty easy to reverse-engineer the production database schema, including any vendor-specific extensions, into a SQL file that can be used to recreate an empty version of the same database schema. Furthermore, it&#x27;s just as simple with most database engines&#x2F;servers to use an automated tool to compare two schemas and generate a DDL script that will upgrade the old schema to the new schema. It helps immensely because such tools will avoid circular dependencies with foreign keys, etc. that may trip you up if you&#x27;re trying to manually write such upgrades.<p>Other than that, this article is spot-on. The database should be the source of all truth and yes, you need to become proficient in SQL to be a good general application developer.
exabrialalmost 7 years ago
I know in the HN comments they&#x27;ll be a whole bunch of &quot;you don&#x27;t do design when you&#x27;re trying to iterate fast&quot;. I wholly disagree. Design phases expose problems in your thinking that you&#x27;ll be tripping over very shortly, except now you didn&#x27;t waste the time implementing something that wouldn&#x27;t work anyway. Skipping the design phase is &quot;productivity theater&quot;, where it appears you&#x27;re being productive because code is being written, but in reality, you&#x27;re just being busy.
drblastalmost 7 years ago
The database really is an API, in that client code depends on it and its structure will be difficult to change once you go live, whether you want that to be true or not. Writing the API before code that depends on it seems like an inherently good idea to me.<p>You can pretend like you don&#x27;t need to figure these things out first and get away with it for a while, but at some point the lack of a solid foundation is going to bite you.<p>I think all devs should answer the question, &quot;What if they don&#x27;t want to use my UI or interface, or they want to automate something?&quot; That&#x27;s a good scenario to support for any client, including yourself.
评论 #17258681 未加载
Fradowalmost 7 years ago
Perhaps I am too new to back-end development, but what strikes me in this article is the misconception that code-first is going to give you a bad DB schema, and that migrations will loose data.<p>I think part of it could be blamed on the tools: from the code I see, the ORM doesn&#x27;t seem well-built, if it needs that much code to generate a proper DB schema.<p>For the record, I&#x27;m using Django (for about a year, I was developping apps before), and my workflow is:<p>1) code, while thinking about what the DB schema will be as a result<p>2) generate the migrations<p>3) apply them and check if what&#x27;s generated is good and usable when you need to directly query the database (mostly for dev purpose). If need be, either change your code, or even manually modify the migrations. Migrations are code too, they are there to be overriden when required.<p>Sure, your rollbacks can loose data, but only if you don&#x27;t pay attention to them and don&#x27;t change them to avoid that. As a developer, it&#x27;s your responsibility to think about that if it&#x27;s of critical importance to your business.<p>Feel free to enlighten me if something I say seems very wrong, after all I&#x27;m still quite junior at this, and have not experienced huge scales &#x2F; large team.
skybrianalmost 7 years ago
The article seems to advocate generating code from a live connection to the production database.<p>This seems very risky. Typically you don&#x27;t want developers to even have access to the production database most of the time, let alone make it a hard dependency to do a build. Development work should be done using temporary databases.<p>Also, database schema migrations need to be tested somewhere other than the prod database before going live. All the code changes needed (including to applications) should be written and tested <i>before</i> actually doing the migration.<p>But this approach would be reasonable with a bit of indirection: one developer (DBA?) makes a database schema change, resulting in a change to a schema file that&#x27;s checked in (after testing), and code gets generated from the schema file. That way developers all have access to a history of the prod database schema, without access to the database itself.<p>(This is pretty close to how things are done using protocol buffers.)
评论 #17248807 未加载
评论 #17250481 未加载
geebeealmost 7 years ago
Fantastic.<p>There&#x27;s an earlier article from the same blog that addresses this at a deep level<p><a href="https:&#x2F;&#x2F;blog.jooq.org&#x2F;2014&#x2F;01&#x2F;02&#x2F;why-your-data-will-outlast-your-sexy-new-technology&#x2F;" rel="nofollow">https:&#x2F;&#x2F;blog.jooq.org&#x2F;2014&#x2F;01&#x2F;02&#x2F;why-your-data-will-outlast-...</a><p>A couple of fundamental principles here: Systems tend to outlast their developers Data tends to outlast the systems operating on it<p>This is why I&#x27;d almost always go with database first design.<p>As a general rule (and test), a well designed database is useful outside of the context of the application that accesses it for persistence.<p>As an example from the kind of apps I&#x27;ve worked on: suppose you are writing a system for tracking and managing inventory, which will be used through a web interface. If you have direct access to the database, can you get useful information about orders, lead times, existing inventory levels, and so forth? If not, you don&#x27;t have a useful database, and you are at risk. Migrating to a new system, which is probably inevitable down the road, will be very difficult as you no longer have control over your data.<p>Now, apps are useful, and code is useful. I wouldn&#x27;t want to try to code up various supply chain management algorithms in SQL directly. The estimated lead time can be quite a calculation, and yeah, you are probably going to want to write python or ruby or java code to do that. But in the ruby world (with rails in particular), I have noticed databases used almost as a kind of object serialization and persistence, where the information makes almost no sense and can&#x27;t be queried in any meaningful manner through the DB. It is simply used to store and retrieve information that needs to be drawn into and reassembled through various bits of code and configuration files in a rails app before it takes a form that makes any sense.<p>In short, the database has no meaning, almost, outside the rails app.<p>Now, I&#x27;m not saying this to knock rails. I&#x27;ve used it, and I enjoyed the experience. Metaprogramming, in fact, makes it very easy to map an object to a database table without a ton of irritating configuration or extra code. You don&#x27;t have to generate your schema through rails to map models to tables, you can create them independently of the app and use rails to do the mapping for you. Also, it&#x27;s entirely possible to use rails generators to create a very sensible database design. The problem is probably that a lot of people using these generators never considered databases or SQL or schemas as something that had meaning outside the context of a rails app. My guess is that this anti-pattern happens in almost all frameworks that allow developers to quickly push objects with relationships off into a database schema that they never took the time to consider or design. It&#x27;s almost no different than if they serialized the objects for persistence, and used a hash to find and retrieve them.<p>I also think this emphasizes why clarity is so important in code and database design. Sometimes people talk about upgrading an existing code base, and say &quot;well, it&#x27;s already written in [java&#x2F;ruby&#x2F;python&#x2F;...] , so it&#x27;ll be easier if we stick with a [java&#x2F;ruby&#x2F;python&#x2F;...] based framework&quot;. Nah. If the code lacks clarity, it&#x27;ll be hell to port it from python to python. If it has clarity, it won&#x27;t be hell to port it from python to ruby.<p>If you can use your database to answer meaningful questions about your data outside the context of an app, your database is not only useful, it also probably has clarity of design and purpose. If you can easily read your code and know what it is intended to do and why, you have clarity.<p>I&#x27;m amazed with how often developers are willing to sacrifice clarity to meet the demands of a testing framework, or a particular architecture, or to accommodate a cutting edge UI&#x2F;javascript approach. These can be good things, but you <i>know</i> that shit ain&#x27;t lasting, and then, without clarity, where will you be?<p>Well, we all know, the developer who gained experience with the nifty new framework will be: at a new, better paid job.<p>Beyond that... eh, I suppose a code generator from a database can be useful and save you some typing, but if it&#x27;s all cluttered and clunky and lacks clarity I&#x27;d rather just do it myself. It&#x27;s not really that hard.
ngrillyalmost 7 years ago
Really great post. Code generation is seen by many as old school, but this post shows it&#x27;s still well alive. We are seeing a revival of code generation with tools like jooq, Protocol Buffers, static website generator, and some Go tools for example.
lucidguppyalmost 7 years ago
There are some who say that architectures that are able to postpone choice of front end and DB are flexible and well designed.<p>You should design your app to make it desktop <i>or</i> web... and choose any DB you want.
nickpetersonalmost 7 years ago
My favorite version of this, is when a new version of something is greenlit by the business, and a bunch of enthusiastic devs look at the old system and decide to rewrite. Instead of starting with the existing data, and determining how it&#x27;s going to fit in the new design, they just make a new design and figure, &quot;we accounted for most business requirements, so the data should be able to fit...&quot;. Then they set a launch date with the business, and just need to do that data migration. What could go wrong?
crb002almost 7 years ago
This goes with any system. Get the data pipelines hot then drive in the business logic. Otherwise you are surprised with latency and have to do major refactors.