TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Understanding the N + 1 queries problem

111 点作者 ananthakumaran超过 2 年前

14 条评论

saila超过 2 年前
You could probably get this down to two queries, one for posts and one for comments, if you aggregate the vote count when retrieving the comments. I think this is pretty easy to do with most ORMs.<p>You could also get it down to 1 query using SQL. This is one way to do it based on the schema in the article [postgres, not well tested]:<p><pre><code> with latest_posts as ( select * from post limit 3 ), latest_comments as ( select c.*, count(v.id) as votes from comment c left join vote v on v.comment_id = c.id where c.post_id in (select id from latest_posts) group by c.id, c.content ) select p.*, json_agg(c) from latest_posts p left join latest_comments c on c.post_id = p.id group by p.id, p.title, p.content # NOTE: fixed SQL bug noted by @rurabe </code></pre> Off the top of my head, I&#x27;m not sure how you would (or if you could) do this with ActiveRecord, SQLAlchemy, or the Django ORM, but it&#x27;s probably more complicated than just writing the SQL.<p>To be clear, I&#x27;m not anti-ORM and use them all the time, but it really helps to understand SQL well when using them and to know when it&#x27;s appropriate to switch to SQL.
评论 #34224039 未加载
评论 #34223672 未加载
评论 #34223693 未加载
评论 #34223803 未加载
kstrauser超过 2 年前
I really wish this been originally called the “1+N problem”, not “N+1”. That naming makes it much clearer to me.
评论 #34231925 未加载
评论 #34229051 未加载
samwillis超过 2 年前
Something interesting to consider with N+1 queries, these warnings only really apply to remote database servers, as in not on the same machine.<p>If you are using SQLite, or another in process database, N+1 isn&#x27;t an issue at all. So with the increased use of SQLite as an &quot;edge&quot; database it&#x27;s something to consider.<p>&quot;Many Small Queries Are Efficient In SQLite&quot;: <a href="https:&#x2F;&#x2F;www.sqlite.org&#x2F;np1queryprob.html" rel="nofollow">https:&#x2F;&#x2F;www.sqlite.org&#x2F;np1queryprob.html</a>
评论 #34223901 未加载
评论 #34222663 未加载
评论 #34223121 未加载
评论 #34223729 未加载
评论 #34232067 未加载
simonw超过 2 年前
There&#x27;s another option with many databases these days: you can often use aggregation functions to return the related data as part of a single query, even across many-to-many tables.<p>I wrote up how to do that using JSON aggregation functions in both SQLite and PostgreSQL for example: <a href="https:&#x2F;&#x2F;til.simonwillison.net&#x2F;sqlite&#x2F;related-rows-single-query" rel="nofollow">https:&#x2F;&#x2F;til.simonwillison.net&#x2F;sqlite&#x2F;related-rows-single-que...</a>
评论 #34223378 未加载
ydnaclementine超过 2 年前
Would this not be solved with adding `votes` to the `includes`? Something like:<p>```<p>Post.includes(comments: :votes)<p>```<p>Similar stackoverflow: <a href="https:&#x2F;&#x2F;stackoverflow.com&#x2F;a&#x2F;24397716" rel="nofollow">https:&#x2F;&#x2F;stackoverflow.com&#x2F;a&#x2F;24397716</a>
评论 #34224094 未加载
评论 #34223067 未加载
keltex超过 2 年前
I don&#x27;t know Rails Active Record. But if the ORM is anything like others I am more familiar with (Django &#x2F; Python or Linq &#x2F; C#) can&#x27;t you do a join and just have a single query? Or use raw SQL if performance is an issue?
评论 #34222470 未加载
评论 #34222355 未加载
评论 #34222383 未加载
acjohnson55超过 2 年前
I can&#x27;t help but feel this problem is indicative of incidental complexity in how we develop web applications. Not saying the PHP glory days were better, but there&#x27;s something to be said for removing the layers of abstraction between the data and the presentation. Make the database query using SQL directly, and then inject the results into the HTML template to be delivered to the browser. Obviously, there were many issues here, like how easy it was to leave applications open to SQL injection attacks.<p>But it has been interesting to see the tide turn back towards server-side rendering, relying on partial DOM replacement for client-side updates. For web apps that don&#x27;t have massive numbers of UI states (like a document editor), it seems like people are rethinking the wisdom thick client-side JavaScript applications, which seem to be one of the main motivators for REST API layers, and the need to efficiently fulfill N+1 queries.<p>Although, I do remember dealing with the N+1 problem when doing Django server-side apps more than a decade ago, before the dominance of client-side apps. I guess it was more the rise of MVC architecture and the active record pattern (<a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Active_record_pattern" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Active_record_pattern</a>) that brought the N+1 problem, more so than client-side apps.
评论 #34223073 未加载
评论 #34224299 未加载
评论 #34223592 未加载
评论 #34223413 未加载
评论 #34231275 未加载
评论 #34228962 未加载
eloisius超过 2 年前
I wish I&#x27;d had an opportunity to use Phoenix in production before I got out of web dev, because the way the Ecto ORM obviated this entire class of error was beautiful. Instead of lazy loading, there&#x27;s a neat grammar for preloading the entire graph of related records that you want.
pharmakom超过 2 年前
This comes up in GraphQL, not just ORMs. A beautify solution is Facebook’s Haxl. Less beautiful is data-loader.
评论 #34232823 未加载
TexanFeller超过 2 年前
Understand N+1 before you try GraphQL.
funnyfoobar超过 2 年前
Somewhat deviating, but relavent. If we use counter cache that is to keep vote_count on comments table, the include(:comments) solution would work fine.<p><a href="https:&#x2F;&#x2F;scoutapm.com&#x2F;blog&#x2F;how-to-start-using-counter-caches-in-rails" rel="nofollow">https:&#x2F;&#x2F;scoutapm.com&#x2F;blog&#x2F;how-to-start-using-counter-caches-...</a>
pmg102超过 2 年前
We solved the N+1 queries problem where I work by raising the level of abstraction from &quot;queries plus serialisation&quot; to &quot;what shape data is required&quot;. We open sourced the solution at <a href="https:&#x2F;&#x2F;www.django-readers.org&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.django-readers.org&#x2F;</a>.
adamzapasnik超过 2 年前
This is what I struggle a lot with in Rails.<p>No good, community backed serialisation gem. AMS is a mess, other ones are not maintained. And I&#x27;m not a fan of JSON api spec&#x27;s serialisation either.<p>But also AR doesn&#x27;t have any easy tools to construct complex queries&#x2F;multi queries. It works for basic and medium stuff, but even this very common count problem is a disaster to deal with. Sure. you can use Arel and some other gems, but these aren&#x27;t good solutions for someone that wants to get things done. Makes me wonder how others deal with these problems tbh.
评论 #34223233 未加载
评论 #34223656 未加载
bfung超过 2 年前
&gt; breadth-first loading. The ideal solution requires us to load the data in a breadth-first approach, but unfortunately, this is harder to write because it does not compose well.<p>The author finds the simplest and efficient solution, but continues to over engineer for blog content :P<p>“Composing” is overrated in this case.
评论 #34226987 未加载