Systems Design for Advanced Beginners

As a solo tech founder of several sites, I’ve had the pleasure of digging into each of these problems and more over the past few years. Some topics worth adding to the list for consumer sites:<p>1. Prototype and benchmark each of your stack pieces before you pick a stack. It is far easier to fix architecture mistakes when you don’t have 10000 users expecting overnight customer service. If you are using new technology, find good open source products to see how they’ve structured their projects. Your architecture, designed for speed and experience, will be your key differentiator. Inherent speed at core task, design to user needs, and name choice are the 3 musketeers of a solid growth.<p>2. Prepare for abusers to attack your system from every direction, especially if you enable users to publish content under your domain. You will see bots looking for Wordpress installations, users trying to fill content with SEO links, users trying every hacking vector known to market. Collect known vectors and test for them and never refuse a legitimate bug bounty request.<p>3. There is an eternal debate about the trade off between building a quality product at start or opening early to get user feedback before you get too deep into features. There is merit to both choices as a solo founder. The moment you open the gates to users, your ability to make changes comes with very high friction. With time, trying new features becomes a tremendous luxury hidden under bug requests, roadmap, customer service replies, etc. Build your biggest riskiest assumptions first.<p>4. Testing is hard as a solo founder. Selenium is your friend. If you don’t spend time with them, your users will take that time in multiples after a mistake.<p>The best way to learn is to come up with a product you really want and build it in your own time. You can test launch in a weekend.<p>When I started launching consumer sites solo as an engineer, I went from a tight specialization to being unafraid to try any tech if it gives me an advantage in solving a problem. Once you’ve simulated enough problems and dealt with the consequences of your choices personally, you can play that 10 level chess game with the architecture of each new feature much faster.

Thought I&#x27;d post this here since I imagine there&#x27;s a strong cross-over of interest:<p>If you&#x27;d like a guided video code-tour of how all the pieces of a production web-app fit together, I publish detailed weekly screencasts showcasing the code, systems, and architecture behind Oxbridge Notes, the business that&#x27;s supported me for the past decade.<p>So far I&#x27;ve covered:<p>— software dependency vetting<p>— data integrity systems (constraints, foreign keys, transactions etc.)<p>— integration testing systems<p>— trade-offs in software quality between customer-facing and admin areas<p>— softer stuff, like designing for SEO (marketing ease is a critical part of any system I design, as an indie-hacker)<p><a href="https:&#x2F;&#x2F;www.semicolonandsons.com&#x2F;series&#x2F;Inside-The-Muse" rel="nofollow">https:&#x2F;&#x2F;www.semicolonandsons.com&#x2F;series&#x2F;Inside-The-Muse</a>

They have this diagram:<p><pre><code> +-----------+ +--------------+ +-----------------+ |Web Browser| |Smartphone App| |Client Libraries&#x2F;| +-----+-----+ +------+-------+ |Other API code | | | +-------+---------+ | v | | +-----+------+ | +---------&gt;+ Steveslist +&lt;-----------+ | Servers | +------------+ </code></pre> Why not this?<p><pre><code> +-----------+ +-----------------+ +--------------+ |Web Browser|--&gt;|Client Libraries&#x2F;|&lt;--|Smartphone App| +-----------+ |Other API code | +--------------+ +-------+---------+ | v +-----+------+ | Steveslist | | Servers | +------------+</code></pre>

Nice tutorial! One small nitpick:<p>&gt; Calculating a hash value from an input is computationally very easy, but reversing the transformation and recovering the original input from its hash value takes so much time and computing power that it is, practically-speaking, impossible.<p>The above is true for encryption but not for hash codes. Recovering the original input from a hash code is not just practically impossible; it&#x27;s <i>provably</i> impossible -- even with an infinite amount of computing power -- because in general a hash code contains less information than the original text.

&gt; How do they store their data?<p>SQL.<p>&gt; How do their different applications talk to each other?<p>Proprietary APIs.<p>&gt; How do they scale their systems to work for millions of users?<p>T H E C L O U D<p>&gt; How do they keep them secure?<p>They just... don&#x27;t.<p>&gt; How do they make sure nothing goes wrong?<p>They just... don&#x27;t.<p>&gt; What are APIs, webhooks and client libraries, when you really get down to it?<p>Easily outsourced to India.

I have been doing a greater part of the things elaborated in the post as a solo founder&#x2F;freelance software person. Albeit the scale of things I need to handle are relatively small.<p>What kind of a job profile should I be looking at if I am in a position where I absolutely need to have one?<p>I don’t slot particularly well in any one thing I feel, neither a great developer nor a great systems person. And I did do a bunch of recruiting and tech consulting too:

This basically outlines my current job. Huge system with many, constantly moving&#x2F;upgrading parts and services across teams, all while utilizing dozens of internal services, tools, and navigating corporate policies. I’m thriving in it, but totally recognize it’s a steep learning curve and takes longer to onboard newcomers.<p>Eventually you get to enjoy deprecating old services as much as building new ones, simply because you never have to teach others about them again.

&gt; Steveslist currently has a very simple and slightly fragile cron setup. We have a single “scheduled jobs server”. We use crontab on this server ... This setup isn’t scaling very well ... We’re considering setting up a new system using a modern tool like Kubernetes.<p>Do people use Kubernetes for running scheduled jobs like this? It seems like it&#x27;d be overkill but in saying that I&#x27;m not sure if I know of anything that can be used for running scheduled tasks in a reliable and observable way that&#x27;s scalable. Maybe Jenkins?

How can I gain practical experience in these things? The jobs I&#x27;ve had mainly revolve around adding new features, not doing any of the things described in the article. Does working at bigger companies actually give you experience with this?

Regarding DB backups, it would be useful to add that it is not enough to simply make backups, one should also test that the backups are actually restorable.

Just recently started messing with AWS Amplify. It&#x27;s crazy how easy it makes basically all this stuff. It really makes it simple to be a solo founder and manage every piece of the stack.

Title corrected - Web application system design for advanced beginners

I love the phrase &quot;Advanced Beginner&quot;

Don’t forget a system for organizing contributions to all this code, a system to ensure compliance with the many legal regimes “Steveslist” is offered in, a system for onboarding new hires, a system for expanding market share, a system for keeping money and reporting on it to wouldbe investors, and a system of systems designers that know how to respond to changing and ever expanding requirements.<p>Don’t sell yourself short: there are systems everywhere you look, not just where there’s convenient precedent to make a web service.

This is the best thing I have read in 2020. As a bug bounty Hunter this is insanely helpful to me .

&gt; To do this, we need to write database queries that aggregate over the entirety of our data. We don’t want to run these queries against our production SQL database, because they could put an enormous amount of load on it. We don’t want a huge query issued by an internal analyst to be able to bring our production database to a grinding halt<p>What kind of query would you have to write to bring down a production db? What makes a solution like hive much better - I guess its optimized for this?

Systems Design is a formidable topic because it can go in so many directions. This is a good guide. Might I also recommend something that includes caching, partitioning, indexing, and NoSQL vs SQL. More detail can be found here <a href="https:&#x2F;&#x2F;www.educative.io&#x2F;courses&#x2F;grokking-the-system-design-interview" rel="nofollow">https:&#x2F;&#x2F;www.educative.io&#x2F;courses&#x2F;grokking-the-system-design-...</a>

&gt; We’re considering either splitting up our cron jobs into multiple servers, or setting up a new system using a modern tool like Kubernetes.<p>:facepalm: beanstalkd can handle a pretty massive amount of jobs before you need to start worrying about scale. I love the huge jump from simple to complex.

Really like the prose&#x2F;Adventure-like writing style of this &quot;tutorial&quot;!

&gt; database engines that are quick at small queries are typically unacceptably slow of answering giant queries<p>Even with proper indexing? I haven&#x27;t seen this issue with Postgres but maybe I wasn&#x27;t working on large enough data sets.

off topic: I like the simplistic design of this website. Can someone please help me understand if this is hosted on WordPress or something like that?

This is amazing. Thank you Robert.

Great post! Thanks for sharing.

Great post!

anyone else just getting a blank page here?

no DDD?