TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

The API database architecture – Stop writing HTTP-GET endpoints

93 pointsby fzeindlabout 1 year ago

30 comments

CafeRacerabout 1 year ago
In reality postgREST sucks... it&#x27;s fine for simple apps, but for something bigger its pain in the butt.<p>* there is no column level security – e.g. I want to show payment_total to admin, but not to user (granted a feature for postgres likely). * with the above, you need to create separate views for each role or maintain complex functions that either render a column or return nothing. * then when you update views, you need to write sql... and you can&#x27;t just write SQL, restart server and see it applied. You need to execute that SQL, meaning you probably need a migration file for prod system. * with each new migration it&#x27;s very easy to loose context of what&#x27;s going on and who changed what. * me and my colleague been making changes to the same view and we would override changes from each other, because our changes would get lost in migration history – again it&#x27;s not one file which we can edit. * writing functions in PlSQL is plain ass, testing them is even harder.<p>I wish there would be some tool w&#x2F; DDL that you can use to define schemas, functions and views which would automatically sync changes to staging environment and then properly these changes on production.<p>Like when you can have a flask kind of app build in whatever metalanguage w&#x2F; ability to easily write tests, then and only then postgREST would be useful for large-scale systems.<p>For us, it&#x27;s just easier to build factories that generate collection&#x2F;item endpoints w&#x2F; a small config change.
评论 #40322972 未加载
评论 #40332171 未加载
评论 #40321453 未加载
评论 #40328905 未加载
评论 #40322601 未加载
评论 #40322859 未加载
评论 #40326203 未加载
评论 #40325532 未加载
评论 #40321776 未加载
评论 #40350380 未加载
评论 #40325329 未加载
评论 #40325545 未加载
评论 #40321896 未加载
评论 #40324486 未加载
languagehackerabout 1 year ago
This doesn&#x27;t seem like a good idea when it comes to consistently hardening an access layer. You mean I get to double the ops maintenance cost of my existing service by adding a new one? You mean I need to figure out how to set up RBAC, rate limiting, logging, and error handling in two places instead of just one?<p>By and large, opinions in API design has suggest against directly mirroring table structure for quite some time. The reasons are many, but they include things like migrating data sources, avoiding tight coupling with the database schema, and maintaining ultimate control over what the payload looks like at the API layer. Just in case you want to do something dynamic, or hydrate data from a secondary service. And if you still want to generate you response payloads like they came straight from the database, there are plenty of code generation or metaprogramming solutions that make providing access via an existing API layer quite simple.<p>This solution seems simpler only because it ignores the problems of most practical service-oriented API architectures. If end users need a direct line to the database, then get your RBAC nailed down and open the DB&#x27;s port up to the end user.
评论 #40325118 未加载
评论 #40321538 未加载
评论 #40326208 未加载
deweyabout 1 year ago
I don&#x27;t understand what&#x27;s the benefit of adding additional third party code into the hot-path. Adding yet another endpoint is hardly a lot of work, sometimes even auto-generated and has the benefit of being available in your existing monitoring &#x2F; instrumentation environment already.<p>Also what about caching if people can &quot;craft&quot; random queries and send them straight to your PG instance?
评论 #40321053 未加载
评论 #40325136 未加载
评论 #40326227 未加载
评论 #40322866 未加载
_AzMooabout 1 year ago
Coupling your API to your database schema is a bad idea. Once you have clients consuming that API you can no longer make changes to your database schema without also updating all of those clients and coordinating their deployments. Advice like this reads like it&#x27;s coming from somebody that&#x27;s never stayed in a role long enough to deal with the consequences of their actions.
评论 #40326218 未加载
评论 #40325527 未加载
评论 #40325539 未加载
评论 #40325935 未加载
wrsabout 1 year ago
Well, the key thing here is writing SQL views to do the access. Once you’re willing to do that, it’s a fairly minor distinction between using PostgREST or writing a thin, possibly even generated, API layer. But that’s exactly what people are typically not willing to do.
评论 #40322043 未加载
xwowsersxabout 1 year ago
&gt; Data retrieval generally does not require any custom business logic, while data-modifying requests do<p>This just doesn&#x27;t seem to be the case in my experience, at least not in most cases. Perhaps for internal services which are completely locked down and you can freely just expose the data via a REST API, but for public facing services I just haven&#x27;t found this to be the case in practice, except in extremely limited circumstances.
评论 #40326285 未加载
martinbaunabout 1 year ago
Very interesting, I knew about PostRest, but never checked it out. A lot of the boilerplate can probably be cut away with using PostRest.<p>Just wondering, what about security and especially if someone decides to DDOS your server with high-load queries? Do you try to filter&#x2F;block those with NGINX?
评论 #40321264 未加载
conqrrabout 1 year ago
Depends on who is consuming your API. Are they internal, maybe ok. Anything else, you likely need a transformation layer and also take into account Auth, rate limiting etc. PostgREST sounds like giving read only access to your DB and I doubt it would be as customizable.
评论 #40326229 未加载
评论 #40325167 未加载
javcasasabout 1 year ago
I have a mixed real-world experience with PostgREST&#x27;s friend: Postgraphile (I&#x2F;E like PostgREST but generate GraphQL instead of REST).<p>It&#x27;s great, but some tools are a bit rough, and people hate you for forcing them to write 8 lines of SQL.<p>The great: authentication works (both internal and using an external service like Okta), most (if not all) standard operations work: get, post, put, patch, delete, filter, sort, select specific columns, autogenerated doc. You have better RBAC than most (if not all) web frameworks out there.<p>The good: proper transactions everywhere, especially if you use a database-backed queue. Use can use views to evolve your interface independent of your data. You keep data consistency at all times.<p>The bad: PostgreSQL permissions can be hard, especially the permissions of views that expose underlying data, and around different schemas. Triggers can get complicated. You need a good tool to evolve your data, and most likely a different tool to evolve your views&#x2F;stored procs, because they change way more often and for different reasons.<p>The horrible: people. They see SQL and they hate it. They rush to replace 8 lines of SQL with 300 lines of ORM and call it an improvent because there is no longer SQL. &quot;Remove SQL&quot; becomes an objective. It doesn&#x27;t matter if they replace it with plague-induced gonorreic syphillis, they think it&#x27;s better. They see a trigger that inserts into a SQL-based queue, now they try to introduce a redis queue, backed by nothing (not even writing to disk), and three endpoints and functions to create and manage it. They claim it to be better, even now that the three functions are unrelated and harder to track.<p>So well, would I do it for my personal projects? Totally. Would I do it in an enterprise project? Nah, they don&#x27;t deserve such an improvement. Let them have their millions of lines of code to replace thousands of SQL.
评论 #40327323 未加载
esafakabout 1 year ago
The title does not reveal that the suggestion applies only to Postgres. It is a bad idea to tie your architecture to a specific technology.
评论 #40321552 未加载
评论 #40321288 未加载
评论 #40325185 未加载
评论 #40325188 未加载
koromakabout 1 year ago
When is an app ever this simple? 95% of the time, you&#x27;re transforming or combining data in the backend, between the client and the DB. Tables are never so pristine.
评论 #40325190 未加载
joshstrangeabout 1 year ago
I have seen and tried enough tools like this to know I should run in the other direction. For rapid prototyping this might be acceptable, as long as you plan to replace every endpoint before launch or maybe for a fully internal tool it would be ok.<p>Security is the problem and no, I don’t want to create a bunch of views to attempt to get it right. Different users have different sets of permissions that give them access to different parts of the data in different contexts. You will pull every one of your hairs out trying to make that work with something like this.<p>I’m not trying to be mean but I find things like this or even Firebase-style tools to be massive foot-guns. Sure, if you get the permissions&#x2F;visibility perfect it might work for you (at this point in time, good luck as you modify it over time) but why take that risk? It’s not like CRUD endpoints are hard to write and I greatly prefer having full control over what I allow in and out of my system in code. That lets me keep all my auth&#x2F;visibility rules in one place instead of spreading them out over multiple systems, which again is foot-gun.<p>I find “I want a tool that does everything for me”-type thinking and “look, it’s magic and it just works”-type tools to be something junior devs flock to (myself included years ago) before realizing they have given up all the control for something that’s really not that hard to do yourself. Same arguments for if you like key-value&#x2F;document-based data stores because there is no schema. There are valid reasons to use both types of data store but if you reason is “this way I can easily change my schema” then I question your ability to write stable systems.
评论 #40329603 未加载
schindlabuaabout 1 year ago
I&#x27;m not a big boy architect and I write small software. To me it seems like &quot;modifying microservices&quot; usually want to read from the database aswell and when I have all my DTOs in place and everything I might aswell implement a GET endpoint?<p>Is this just for performance considerations? When you have lots of reads and maybe few writes so you need an extra service to handle the load?<p>Can someone give a practical example?
评论 #40326236 未加载
dicyteaabout 1 year ago
<a href="https:&#x2F;&#x2F;www.postgrest.org" rel="nofollow">https:&#x2F;&#x2F;www.postgrest.org</a> links me to a weird and broken WordPress news site.<p><a href="https:&#x2F;&#x2F;postgrest.org" rel="nofollow">https:&#x2F;&#x2F;postgrest.org</a> on the other hand links me to the correct documentation site for PostgREST.<p>Anyone else having the same problem?
评论 #40322786 未加载
SJC_Hackerabout 1 year ago
Don&#x27;t see any support for CTEs, window functions or joins. Admittedly joins should probably be handled as views, but window functions and CTEs can be very useful in some circumstances.<p>It does seem like for basic CRUD this should be all you need. But you will probably run into situations where it can&#x27;t handle something, and then you have to write your own endpoints anyway.<p>Writing your own endpoints also allows more flexibility if yu want to do some post&#x2F;pre-processing steps.
评论 #40326254 未加载
评论 #40326414 未加载
rblatzabout 1 year ago
Yet another solution that makes easy things easier, while making just about everything else significantly harder if not impossible. RBAC, monitoring, telemetry, merging data from multiple sources, migrating between data stores, database structure optimizations, caching... These are just of the top of my head of issues that this is likely to make harder, all to prevent writing some of the most simple code you can write.
评论 #40322499 未加载
评论 #40325200 未加载
lyu07282about 1 year ago
Well, database introspection isn&#x27;t new, we could&#x27;ve done that decades ago, but there are good reasons for the layer of abstraction.<p>Also if you go down this route, I think Postgraphile is a much better realization of that idea (uses GraphQL, not REST): <a href="https:&#x2F;&#x2F;www.graphile.org&#x2F;postgraphile&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.graphile.org&#x2F;postgraphile&#x2F;</a>
评论 #40325219 未加载
评论 #40326237 未加载
hervemabout 1 year ago
&gt; I call this approach the &quot;API database architecture&quot;<p>Why people can&#x27;t search for an existing term before creating a new one, it just add confusion into bucket which already contains &quot;DB as API&quot;, &quot;DB over API&quot;, &quot;DB 2 API&quot;, &quot;DB 2 REST&quot;, &quot;DB low-code API&quot;
fire_lakeabout 1 year ago
How can you apply permissions logic that is not easily expressed in Postgres itself? That’s the main reason I want to build an API in a “proper” programming language.<p>I suppose I could build that in front of Postgrest but why add latency?
评论 #40328576 未加载
vbilopavabout 1 year ago
Here&#x27;s an alternative I&#x27;ve built for myself <a href="https:&#x2F;&#x2F;github.com&#x2F;vb-consulting&#x2F;NpgsqlRest">https:&#x2F;&#x2F;github.com&#x2F;vb-consulting&#x2F;NpgsqlRest</a>
jurschreuderabout 1 year ago
The GET ones are actually easiest to write, like 30 seconds or something.
评论 #40326532 未加载
评论 #40326427 未加载
DeathArrowabout 1 year ago
I am not convinced. In an endpoint I can do much more than just interrogating the database. And I usually do.<p>It might work for simple cases, but then why not also provide POST, PUT, DELETE, PATCH?
评论 #40326406 未加载
sufehmiabout 1 year ago
Note: Exposing your database directly to the Internet &#x2F; external access should always be considered a bad idea &amp; and as a last resort.
评论 #40326424 未加载
graphememesabout 1 year ago
this is such a bad idea, ya lets just increase our infrastructure burden 5x to not write 3 get requests
评论 #40326420 未加载
socketclusterabout 1 year ago
It&#x27;s interesting reading this because I implemented a Node.js solution for this problem years ago but it fell on deaf ears. GraphQL was getting all the attention at the time.<p><a href="https:&#x2F;&#x2F;github.com&#x2F;socketcluster&#x2F;ag-crud-rethink">https:&#x2F;&#x2F;github.com&#x2F;socketcluster&#x2F;ag-crud-rethink</a><p>I wrote it for RethinkDB but it could be adapted to any database as it doesn&#x27;t rely on changefeeds.<p>I then ended up building a complete serverless solution around it: <a href="https:&#x2F;&#x2F;saasufy.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;saasufy.com&#x2F;</a><p>It borrows many concepts from REST but works over WebSockets. Why WebSockets? Two major reasons:<p>- It had to support real time subscriptions&#x2F;updates so that the views could automatically update themselves when data changed (e.g. with concurrent users). I didn&#x27;t want to force the developer to manage channels manually as it can be a major headache to get the subscription order right and to recover from disconnections without possibility of missing any update messages. Also, my library SocketCluster already supported client side pub&#x2F;sub with clustering&#x2F;sharding on the back end so I wanted to leverage that mechanism.<p>- WebSocket frames are tiny and don&#x27;t have all the overhead of HTTP requests so it&#x27;s possible to have field-level granularity which is important for avoiding resource update conflicts. This is something that the GraphQL developers also figured out at some point. But the challenge with true end-to-end field-level granularity is that loading a single resource would require a potentially large number of requests to be made; hence HTTP requests are not suitable for this (imagine HTTP headers containing cookies being sent for every single field of a resource), however, WebSocket frames are ideal for that as they have tiny headers. You can handle maybe 100 WebSocket frames for the same cost as a single HTTP request.<p>End-to-end, field-level granularity is powerful as it allows subscriptions to be set up automatically per-field and access control can be enforced automatically at both the resource and field level (for each of create, read, update, delete and subscribe operations). It&#x27;s also very useful for caching because different views may display different fields of the same resources with some shared fields so caching field values allows different views to share cache.<p>The overarching philosophy behind end-to-end field-level granularity is that it allows the system to treat each field of a resource as an independent entity with its own subscription&#x2F;synching mechanism, access control and cache.<p>SocketCluster was designed to facilitate extremely cheap pub&#x2F;sub channel creation (both in terms of CPU and memory) and automatic cleanup so it seemed like a good use case to build on top of my existing work. The code of ag-crud-rethink is quite simple... Only about 1.5k lines of code and probably could have been a lot smaller without all the bells and whistles.<p>The view mechanism also supports real time updates. It can be thought of as a parameterized collection. You define a &#x27;view&#x27; with one or more parameters to control the filtering and&#x2F;or sorting (though you can construct essentially any query on the back end). The idea is that the parameters for the view are provided by the client. This means that you can represent any view of a collection as a simple string which can be used as a channel name; this is useful for efficiently delivering real time updates since we only want to deliver resource change notifications to views which include that resource. During writes, field names can be cheaply matched against view parameters to decide which instances of the views need to receive the notification. Only clients which are looking at an affected view right now will receive the notification.
tarasglekabout 1 year ago
Can you please add rss to your blog?
评论 #40326244 未加载
dventimiabout 1 year ago
The particular tool Fabien recommends may be &quot;PostgREST&quot;, but the general approach is &quot;API database architecture&quot;, which has been adopted by a number of other tools as well: PostGraphile, Prisma, and Hasura come to mind. There is a lot of criticism of this approach in the comments here, but they exhibit a lot of repetition, so let me consolidate them--along with my responses--in one place and be done with it.<p><i>You should never expose your database.</i><p>Let me stop you right there. Please don&#x27;t tell me what I should do, especially if you don&#x27;t know what my circumstances are. You&#x27;ll get a less frosty response if you make your criticism impersonal and frame it in terms of trade-offs.<p><i>Fine. &#x27;One&#x27; should never expose their database.</i><p>Unless it&#x27;s accompanied by reasoning and evidence, I&#x27;m going to regard criticisms like these as received wisdom.<p><i>OK. One should never expose their database because of security concerns.</i><p>PostgREST (for instance) addresses these concerns through a combination of web tokens, database roles, database permissions, and row level security. Other tools (Hasura, PostGraphile) are similar. If this strategy is inadequate, well that&#x27;s an interesting topic. Please elaborate. If there&#x27;s valid criticism, perhaps there&#x27;s something the PostgREST team can do about it.<p><i>Also, one should never expose their database because the API should be decoupled from the data model.</i><p>Again with the received wisdom.<p><i>Fine. The API should be decoupled from the data model because of good reasons, like freeing the underlying data model to change without breaking clients.</i><p>With this approach, the API can be decoupled from the data model by using schema, views, and procedures in SQL. If that&#x27;s inadequate, again that&#x27;s an interesting topic. Let&#x27;s hear more.<p><i>If the database is going to be exposed, then just go all the way and open up the database port.</i><p>Remember the thing above about &quot;circumstances&quot;? Sometimes, the circumstances don&#x27;t allow this. For example, sometimes circumstances demand an HTTP API. That&#x27;s going to be difficult for many databases without a little help. That&#x27;s where things like PostgREST comes in.<p><i>This approach doesn&#x27;t save much effort because creating APIs in code is so trivial that it can even be automated.</i><p>If it&#x27;s trivial, then why pay an expensive developer to do it? If it can be automated, well that&#x27;s what exactly what PostgREST is doing: automating APIs.<p><i>This approach adds another layer</i><p>No, this approach replaces one kind of layer with a different kind of layer. It replaces (for example) Spring Boot with PostgREST. If Hibernate is also being used, then arguably PostgREST is replacing two layers.<p><i>ORMs are bad</i><p>I&#x27;m sympathetic to that point-of-view but regard it as a <i>non-sequitur</i> since PostgREST isn&#x27;t an ORM. Neither is Prisma, for that matter, despite what their marketing material says.<p><i>REST is bad</i><p>Traditionally, APIs of any stripe were difficult to code by hand. Web APIs tended to be REST APIs for a long time. Ergo, REST APIs were difficult to code by hand, not because they were REST but just because that was the nature of things. Consequently, REST APIs got a bit of a bad rap. The ease of providing REST APIs with PostgREST, however, warrants revisiting that criticism &quot;REST is bad.&quot;<p><i>But this leaves me no place to put my business logic</i><p>Well, some kinds of business logic (input validation, sequencing, and data transformations) can be handled in the database. Other kinds are more challenging (side-effects, for example). I&#x27;d love to hear the details.<p><i>Even if the general approach of &quot;API database architecture&quot; can be good, why use PostgREST instead of writing your own framework which is more database-agnostic</i><p>Two reasons: One, I don&#x27;t want to write the code and the PostgREST developers are probably better than I am anyway. Two, I don&#x27;t need the database agnosticism. People who <i>do</i> need support for another database might consider Prisma or Hasura. If that doesn&#x27;t fit the bill then yes, in that case custom code probably is in order.<p><i>Building applications in the database, in SQL, feels awkward.</i><p>Many things feel awkward at first.<p><i>Building applications in the database just isn&#x27;t how it&#x27;s done at my organization.</i><p>Fair enough, but that&#x27;s a <i>social</i> obstacle to this approach, not a <i>technical</i> obstacle.
tuetuopayabout 1 year ago
Oh great, now you exposed your database schema to outside consumers. Guess you&#x27;re not making any migration then :)
评论 #40326243 未加载
评论 #40322099 未加载
评论 #40325201 未加载
tills13about 1 year ago
Oh this is an ad
mannyvabout 1 year ago
Yeah, let&#x27;s put another server that can fail in the path of your product because more failure points are better.<p>But at least you&#x27;re not providing your user a direct link to your database, which is almost always a bad idea.
评论 #40325206 未加载