Poor Man's Scalability

Cross posting my reply from another site:<p>&#62; We were now rendering our site index in 1.5 seconds or less, compared to 3 seconds before.<p>This is the real WTF. What kind of queries take three seconds to run, that is ridiculous. Even with indexes 1.5 seconds is insane.<p>Deciding against premature optimisation isn't permission to throw away good, efficient design. It's clear from the rest of the article that these guys really have no idea what they're doing. Quoting the results of all their optimisations:<p>&#62; In the end, we did not go down. The last round of load tests with httperf showed us handling 200 GETs a second indefinitely, with an average response time of about 1.2 seconds. We served thousands of requests without even a hiccup, and we did it without spending any money.<p>This is really poor performance, and it shows in the site when you change page or select a different category/filter. I'm guessing that they also chose a nosql solution for the wrong reasons, as it seems everything there could be done much faster/better by using an ACID compliant database.

Thanks to the author for taking time to write up his experience. That said, this is really basic stuff. How in the world could a home/index page have so many queries that it took 9 seconds? Oh, no indices... Then how could you have built a DB-backed web app and not considered indexing? A better title for the post might be "Scaling Mistakes I Made While Building Our Website and How I Fixed Them".

It's fairly basic stuff this, but important none the less.<p>Kudos for recommending Yslow (or PageSpeed) - it's indeed brilliant. You didn't mention much about http caching, which I think is probably as important as the other things you mentioned; Not only does it improve performance for the user, but it also reduces load on your server and it enables you to put up an edge side cache for extra performance.<p>As for the problems mentioned with the load balancer - You could have simply provisioned a new instance and installed your own load balancer. There's dedicated packages like haproxy, but you could also just put up nginx or lighthttpd. This machine can later double as your edge cache (Squid).<p>Btw. 1.5s to <i>render</i> the front page? That's way beyond acceptable, in my book. But I suppose it depends a bit on what the web site does.

Because i'm in a particularly pedantic mood: I don't think ELB (Amazon's load balancing) is free, which is a little misleading.<p>In fact:<p>* $0.025 per Elastic Load Balancer-hour (or partial hour)<p>* $0.008 per GB of data processed by an Elastic Load Balancer<p>Source: <a href="http://aws.amazon.com/ec2/pricing/" rel="nofollow">http://aws.amazon.com/ec2/pricing/</a><p>Obviously these are problems loads of people face / hope to face, so it would have been nice to get some more 'meat' in this post.<p>Edit: formatting

No CDN? Adding CloudFront is really simple and the costs are insignificant.

Your story closely resembles the challenges we had scaling sommelierwine.ca before a media event at our latest launch location. I knew that our site would receive an increase in traffic, and the increase would only be temporary.<p>As a rails site hosted on Heroku, this gave us the ability to scale our site using the gambit of widgets that Heroku offers; but, we didn’t want to spend the money and cheat ourselves from the satisfaction of scaling our site.<p>Using New Relic (a gem available on Heroku) I identified our first performance bottleneck - database queries. Certain queries were taking over 30 seconds to complete! These database queries all involved a table of BLOBs (on the order of 20MB) that needed to be searched and sorted. I tried adding indexes but that only marginally reduced the query time. The solution we found included moving the BLOBs to their own table while keeping an index to them in a table of their attributes [1]. Doing this, we were able to reduce query times down to less than a 100ms.<p>The remaining bottlenecks we found using YSlow has helped us reduce the overhead in loading pages substantially.<p>Even though we were able to weather the storm of visitors to our site, we did leave some tasks for the next one including implementing caching, background processing, and remote file storage. All in time.<p>Does anyone have other wallet-friendly Rails/Heroku scaling stories share?<p>[1] This is database 101 stuff - a class I never took.

The next step after ensuring you're doing the basics (cdn, minimizing data over the wire, etc) for us was to throw the entire site behind varnish. Set a "don't cache" cookie if users login, add a comment, etc. so that their changes are visible immediately.<p>We took a slow site running on hardware, put it on EC2 (slower hardware obviously), and with varnish are saving about 60% cost of hosting, and decreased page load time on some of the popular pages by upwards of 50%, all for minimal engineering effort.

This seems like basic stuff. I'm more interested in the story about how you got this to the top of the front page.<p><i>Edit: Less obnoxious.</i>

Cody, just one note on your website - I couldn't tell at all what Famigo offers without clicking on something. Instead of having "Famigo helps families find and manage mobile games and apps" in a hover-over pop-up, have a similar tagline somewhere people can see easily.

"Moving the Javascript to the bottom" - wouldn't the best thing to do would be to invoke the JS from onLoad? And put the code in a separate minified JS file, so it can be cached?

Nice writeup and valuable experience. Just curious why no extra money was spent? Spinning up more EC2 instances costs money, right?

With the rise of cloud services, and their relative inability to serve naked domains, there are a couple of easy fixes for this.<p>The first is to create a vhost in your webserver that serves an index.htm page on your NON-www url. The contents of that page would simply redirect users to the www.yourdomain url.<p>If that seems like too much work, an even easier fix (truncated for brevity) is to do something like this:<p><pre><code> var url = window.location.href; if(url.indexOf("http://www") != 0) { window.location.replace("http://www.yourdomain/"); } </code></pre> You'd obviously want to capture the remainder of the querystring and add it in, but that's the general idea.