I am planning to build something like Reddit for our Region (it will mostly be about regional topics).<p>I found out Reddit is getting 250M comments per month and though we will never be this big, I am wondering what technology stack we should use to build Reddit to overcome scaling challenges.<p>I believe the DB will be Mongo but what can be the right web stack?
Maybe spez can chime in, but I believe the first Python re-write of Reddit 1.0 was in Pylons. The key was pre-render every page, no ajax, just pagination, and sacrifice page "freshness" for stability. Modern equivalent is something like static NuxtJS deployed to Netlify, but with tight re-generation loops. At <1M writes per day, its an interesting challenge if the site is regionally limited, and you are trying to self host and avoid cloud costs. And no one really does old school caching any more as its all React / infinite scrolling. Best of Luck ;)
There are existing open-source clones with sites that are active. What are they doing wrong?<p><a href="https://gitlab.com/postmill/Postmill" rel="nofollow">https://gitlab.com/postmill/Postmill</a><p><a href="https://github.com/ruqqus/ruqqus" rel="nofollow">https://github.com/ruqqus/ruqqus</a><p><a href="https://github.com/Phuks-co" rel="nofollow">https://github.com/Phuks-co</a>
As far as I know, reddit uses postgresql with heavy partitioning.<p>I personally would stay far from MongoDB because of personal preferences and try something else, like Citus Hyperscale for Postgresql.<p>Then, I would make a prototype using hasura.io and something like next.js<p>My backup plan would be to go with Ruby On Rails because it's actually still very good.
You need to work out where you sit on the efficiency vs development speed and your architecture.<p>If your focusing on development speed to be able to add features and trying to grow quickly, then your should lean towards where existing skills are, but design it in a scale out fashion. i.e. break out your state from your http layer so you can run as many front end instances as needed to handle the load. Work on getting the data layer optimized as far as data access patterns go, and serve most of your data from a caching layer. Keep in mind that a web app like reddit spends most of its time dealing with your data layer. A Higher performance language won't fix a slow data source.<p>If your optimizing for efficiency (hosting and cost), then it might make more sense to start with a higher performance language, but at that point you likely want to optimize for your hosting environment. Can you make your application highly static? Can you make the pages be generated on change? i.e. new comments / posts go into a worker queue and a static HTML page is generated (possibly with some small dynamic content being pulled in from another microservice). In this architecture the web stack is dictated by your hosting environment. For AWS this would be whatever runs best in lambdas. Something like Cloudflare workers enforce JS or Web Assembly.<p>You also need to ask yourself if you want to develop the code or the site. If your more interested in the application then you should look for something that already exists that is close and see if you can make it work for your needs. The site likely will need a lot of work to attract a userbase, and your likely better focusing on that unless the goal is to develop the software itself.
You should not be thinking about scaling at this point. Just build the thing and go from there. The chances of you getting enough users for 1000s of comments per minute are low. Worry about that problem when you get there, it's a good one to have.