TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

How Instagram Feeds Work: Celery and RabbitMQ

56 pointsby thikonomalmost 11 years ago

6 comments

wuliwongalmost 11 years ago
They are overstated the naivety of the &quot;naive&quot; approach.<p>The article states:<p>&#x27;directly fetch all the photos that the user followed from a single, monolithic data store, sort them by creation time and then only display the latest 10&#x27;<p>That isn&#x27;t how the query would work. It is implying that the query would return all the results, clearly it would only return 10. I haven&#x27;t read all the details, I&#x27;m sure Instagram does something better then this basic SQL now but it is silly in a technical article to overstate the problem in such an obviously incorrect way.
carbocationalmost 11 years ago
In short, the feeds are heavily denormalized and are constructed when new photos are added by people you follow, rather than at request time.<p>They seem to favor using disk space and saving on processing time and memory.
评论 #8059061 未加载
megaman821almost 11 years ago
It seems that these problems would be better solved at the database level. It is just that the open source databases lack good implementations of materialized views and data replication to slaves. Instead you have this more fragile solution involving the Postgres, RabbitMQ, and Redis to accomplish nearly the same thing.
acoyfellowalmost 11 years ago
This post is great.. but its from April 2013. I wonder how their architecture has changed since then?
评论 #8058688 未加载
lobster_johnsonalmost 11 years ago
Were they only using two RabbitMQ brokers? I have a habit of running one per machine, which allows every app to simply connect to localhost; but on virtualized clusters, RabbitMQ gets a network partition almost every other day, and having just two would obviously cut down on the number of partitions.
alttabalmost 11 years ago
If they are using EC2 hosts, I wonder why they don&#x27;t use AWS SQS&#x2F;SNS for their queue systems and Dynamo for their materialized views.
评论 #8058741 未加载