This seems to be the dataloader pattern. There are implementations in many languages, but the idea is that you have a bunch of threads which declare their I/O needs, and then you 1) debounce and merge the requests (uniform access) and 2) cache the results so that later in the graph of calls you don’t need to fetch already loaded data.<p>Here’s one impl: <a href="https://github.com/graphql/dataloader">https://github.com/graphql/dataloader</a>
I see all these comments stating 'oh ORMs are bad' and 'just write some SQL'. Yes, you should probably not be afraid of SQL, and yes, using an ORM for everything is probably not great, and no, ORMs aren't a full replacement for writing SQL occasionally, but taking an extremist pro-SQL point-of-view is not doing any favors to this debate.<p>There are very real reasons why writing raw SQL is a pain. You can make arguments about typesafety of queries, maintainability, mapping into application level data structures/types yadayada. Imho the primary argument is that you cannot efficiently write a query that doesn't suffer from duplication of data. If you have an entity A that has many B's, and that same entity A also has many C's, then if you join on both you now are loading A * B * C rows. This is going to be slow too, and is difficult to detangle.<p>ORMs in their current form are terrible at tackling this problem, and people are right to call them out for that. "Just" lazy loading or "just" eager loading both do not scale, nor does doing firing event listeners on a per-entity basis. But rawdogging SQL is not a viable solution either, and I wish people advocating that would stop this attitude of 'oh you prissy dev using your puny ORM, not willing to write a line of SQL, look at me with my glorious hair on my chest and superior intellect writing pristine queries'.
Jet can automatically load joined objects into embedded Go structs: <a href="https://github.com/go-jet/jet/wiki/Query-Result-Mapping-(QRM)">https://github.com/go-jet/jet/wiki/Query-Result-Mapping-(QRM...</a><p>Depending on what you are doing there might be some duplication that you could remove by creating hash lookups as in this post, but I would reach for Jet first.<p>sqlc supports embedding but not embedded slices?
For detecting rather than preventing duplicates, I'm fond of this pattern in Go:<p>At "entry" to your code, add a mutable "X was called" tracker to your context. Anywhere you want to track duplicates, insert/increment something in that tracker in the context. And then when you exit, log the duplicates (at the place where you put the tracker in).<p>It's reasonably implicit, works for both tracking and implicitly deduplicating (turn it into a cache rather than a counter and voila, lazy deduplication*), and it's the sort of thing that all your middle layers of code don't need to know anything about. As long as they forward contexts correctly, which you REALLY want to do all the time anyway, it Just Works™.<p>*: obviously you can go overboard with this, there are read-after-write concerns in many cases, etc which this article's "prevent by design" structure generally handles better by making the phases of behavior explicit. but when it works, it's quite easy.
If you're using Go, sqlboiler can do this for you in most common cases (e.g., fetch all the Users matching this filter, and for each User, fetch the related Company)<p><a href="https://github.com/volatiletech/sqlboiler">https://github.com/volatiletech/sqlboiler</a>
I've always been sort of fond of 1 + 1. It's too often the case that there's a popular query that doesn't even need the child data to function, and unless you have some elaborate caching mechanism it would be a shame to pay the full cost of the join or however you want to implement it.<p>Making one query that returns the base data and a second that pulls all of the associated data works often enough.<p>Then it's only when you need to pull M individual records and the associated data that <i>might</i> put you into M + 1 queries, if you can't work out client side grouping for some esoteric reason. But you've reduced the exponent of the fanout by 1, which can hold you for a long time. Years even.
To work around N+1 is to write all your database layer functions in form of getFoosByIds(id[]) instead of getFooById(id). This allows you to easily compose the loads when you have subresources. It's similar to what the author is doing, but does not tear apart the subresources from the parent object.<p>Pushing the subresource fetching down to the database requires using JOINs and fails badly when you have multiple one-to-many relations in one fetch.<p>Just do a single, separate query per table.
Ah, I look forward to every brandur post! :-)<p>If he can give up Go, we've got a TypeScript ORM that will de-N+1 basically everything* that is not a paginated/limit-offset query:<p><a href="https://joist-orm.io/docs/goals/avoiding-n-plus-1s" rel="nofollow">https://joist-orm.io/docs/goals/avoiding-n-plus-1s</a><p>This works even in adhoc loops, i.e. if you have a lifecycle hook** of "after an author changes, do x/y/z logic", and you update 100 authors, every SQL operation invoked by those ~100 individual hooks is auto-batched.<p>We've been running this in production for ~4 years at this point, and haven't had an N+1 since then (although we didn't initially support auto-batch find queries; that came later).<p>Of course kudos to dataloader.<p>*everything --> any queries our "find" API supports, which doesn't do aggregates, sums, havings, etc.<p>**lifecycle hooks --> yes, a blessing and a curse; we're always attempting to find better/higher-level abstractions for declaring the intent of business logic, than raw/imperative hooks.
I'm very grateful to this post for introducing me to sliceutils to create a map from a slice. I think that's a very elegant way to create nested models given a parent and child struct.
The problem is not in Go's endless verbosity. The problem is the basic concept of ORMs - that the model you need for efficiently rendering data is the same model that you need for efficiently storing data. ORMs map them 1:1 and that's what results in N+1 queries and all the other problems.<p>Go's endless verbosity and lack of dynamic features is a blessing, not a curse. Because you have to write your own data access layer, you can break this blatantly false assumption that what you need in the UI is the same as what you need in the database.<p>To break the N+1 problem, you can do funky stuff like pull back sub-components as array fields in the main component, or pull multiple result sets, or concatenate the sub-component data into a single text field, whatever your UI requires. Because you're not forcing the databases query to map 1:1 with whatever structs you've got inside your application, you can get creative with how you structure the database storage, and how you query that storage. You can create funcs in the database that do crazy shit with the data to return exactly what the UI requires in a custom struct. Because it all runs on the database itself it's fast and painless. Because you have to manually map all the structs in your application yourself (thanks Go!) then you are not constrained to mirror the database structure in your code. Your UI code does what it needs, your database does what it needs, and you can map between them however you want.<p>ActiveRecord is easy to use, and very concise, which is what it optimises for. Go is not optimising for the same thing. The author is trying to recreate ActiveRecord in Go without realising that ActiveRecord is a straightjacket, and obviously struggling. If you free yourself from those constraints, the world becomes a simpler, better, place.
I think the N+1 problem is overblown. The number of database calls will scale with the volume of data retrieved, but the volume is data retrieved should always be small.
ORMs are such a toxic idea, we'd be better off if we could un-invent them. Look how thoroughly this poor author's mind has been poisoned by long-term exposure.<p>Please get out there and write some SQL. I promise it won't hurt you.
As a side note, I think I am adding “render” to my list of words which are meaningless and therefore banned from using in code:<p>- render<p>- container<p>- context<p>- base<p>- model<p>- action<p>- component<p>- bake<p>- build<p>- generate<p>- view<p>- control<p>- process<p>- resource<p>Any I should add/remove?
everything about this screams wtf to me<p>engineering around deficiencies sometimes yields interesting results but this isn't one of them.<p>I'd say this code is spaghetti.