A few things have caught my attention in your post.<p>Your biggest problem was that the configuration of your services was not sized/tuned properly for the hardware resources you've got. As a result of this your servers have become unresponsive and instead of fixing the problem, you've had to wait 30+ minutes until the servers recovered.<p>In your case you should have limited Solr's JVM memory size to the amount of RAM that your server can actually allocate to it (check your heap settings and possibly the PermGen space allocation).<p>If all services are sized properly, under no circumstance should your server become completely unresponsive, only the overloaded services would be affected. This would allow you or your System Administrator to login and fix the root-cause, instead of having to wait 30+ minutes for the server to recover or be rebooted. In the end it will allow you to react and interact with the systems.<p>The basic principle is that your production servers should never swap (that's why setting vm.swappines=0 sysctl is very important). The moment your services start swapping your performance will suffer so much that your server will not be able to handle any of the requests and they will keep piling up until a total meltdown.<p>In your case OOM killing the java process actually saved you by allowing you to login to the server. I wouldn't consider setting the OOM reaction to "panic" a good approach - if there is a similar problem and you reboot the server, you will have no idea what caused the memory usage to grow in the first place.
1. Reduce keepalive, even with nginx 60 is too much (unless it's an "expensive" ssl connection).<p>2. set <i>vm.swappiness = 0</i> to make sure crippling hard drive swap doesn't start until it absolutely has to<p>3. Use <i>IPTABLES xt_connlimit</i> to make sure people aren't abusing connections, even by accident - no client should have more than 20 connections to port 80, maybe even as low as 5 if your server is under a "friendly" ddos. If you are reverse proxying to apache, connlimit is a MUST.
If anyone owns a blog or site that they suspect <i>may</i> appear on HackerNews (especially if you're posting it), then please take the small amount of time to put an instance of Varnish in front of the site.<p>Then, ensure that Varnish is actually caching every element of the page, and that you are seeing the cache being hit consistently.<p>You should expect over 10,000 unique visitors within 24 hours, with most coming in the 30 minutes to 2 hours after you've hit the front page on HN.<p>You need not do your whole site... but definitely ensure that the key landing page can take the strain.<p>Unless you've put something like Varnish in front of your web servers, there's a good chance your web server is going down, especially if your pages are quite dynamic and require any processing.
I'd argue the opposite of your headline, that this was a very successful launch. Since HN isn't your target audience having your site fail from the traffic was far better than having it fail from a launch in your market. You shook out some important bugs before you lost real users. Plus you got to do this followup which will bring even more traffic.
First off, best of luck with your project. Secondly, kudos on writing the post-mortem, as I know it takes some guts to own a "failure".<p>I think, however, the need to write something like this speaks to an incorrection assumption: you need a "launch". Of course, TC and HN can give you a nice bump in traffic and even signups. However, in the long run, this really doesn't accomplish much for you. It gives you the kind of traffic that will likely leave and move on to the next article, skewing your metrics. There's certainly qualified prospects in there, but it's hard to decipher with all the noise.<p>Again, the concept of a "launch" speaks to poor business models. It really benefits businesses where the word "traction" is more important than "revenue". Build a business that provides a service that others will pay for and grow as fast as the business can bear, bringing in those visitors that are truly valuable to you.
Thanks for this post, there were some nice tips in there. Although, I do have some nitpicking about your writing style. Maybe it's just me, but I found that your use of "+ve" instead of just saying "positive" and of "&" instead of "and" did not have the intended effect of speeding up reading, quite the reverse actually.
Running with swap enabled is a terrible idea. The authors mention how it was only once solr crashed that they were able to actually log in and start fixing problems; having swap means that rather than the OOM killer terminating processes, instead your whole system just grinds to a halt.<p>(it's strange that they recommend enabling swap when they also recommend enabling reboot-on-oom, which is pretty much the complete opposite philosophy)
1. So nginx didn't cache because of cookie?<p>2. Isn't swapping bad? I don't think I've ever had a situation in which swap more than say 100MB was helpful. Once the machine starts swapping, a bigger swap just prolongs the agony.<p>3. If you couldn't ssh, why didn't you just reboot the machine?<p>Edit:<p>1. What did you use for the graphs?<p>2. What is the stack?
Stress test, load test before launch!<p>It doesn take more then an hour, and you quickly know what your upper limits are, and where the bottlenecks are.<p>I use gatling in favor of JMeter:
<a href="https://github.com/excilys/gatling" rel="nofollow">https://github.com/excilys/gatling</a>
I find it very difficult to believe that this person worked on any sort of performance team, given that what they discovered is pretty much "Handling Load 101".<p>Running everything on one box? Using swap? No caching? It's like a laundry list of junior admin mistakes.
This post mortem has me thinking about the best way to handle the situation in which you can't SSH into your server. The OP decided to trigger a kernel panic/restart on OOM errors, but I have a couple of concerns about this approach:<p>* If memory serves correctly, if your system runs out of memory, shouldn't the scheduler kill processes that are using too much memory? If this is the case, the system should recover from the OOM error and no restart should be needed.<p>* OOM errors aren't the only way to get a system into a state where you cannot SSH into a system. It would be great to have a more general solution.<p>* Even if you do restart, unless you had some kind of performance monitoring enabled, the system is no longer in the high-memory state so it will take a bit of digging to determine the root cause. If OOM errors are logged to syslog or something, I guess this isn't a big deal.<p>I suppose the best fail-safe solution is to ensure you always have one of the following:<p>* physical access to the system<p>* a way to access the console indirectly (something like VSphere comes to mind)<p>* Services like linode allow you to restart your system remotely, which would have been useful in this scenario
I clicked on the link to Cucumbertown and was immediately greeted with a picture of Italian seasoned chicken thighs.<p>I think I really like your website. I really like the simplicity of the presentation to the user.
This is a great way to make lemonade out of the lemon of getting hosed by a lot of traffic. Write an informative post-mortem and resubmit! I know I missed the original submission and clicked through to the site, and there you have it. I'd say being humble and trying again is never a bad idea.
Whatever was the <i>real</i> cause for your issues, Linode's default small swap space is a plague. A system starts to misbehave much gently if there is enough swap.
[ not that you asked for it here, but I've got some frontpage UI feedback: ]<p>I think you should put a description up front to describe what Cucumber town is. I think that main image should be a slider with multiple feature images, and I think the Latest Recipes should be the first section after this. Just my 2c!<p>Screen: <a href="http://cl.ly/image/3R2Y131Z433L" rel="nofollow">http://cl.ly/image/3R2Y131Z433L</a>
I have been having some of the same issues on a site I run ( <a href="http://www.opentestsearch.com/" rel="nofollow">http://www.opentestsearch.com/</a> ). Under heavy load solr will grind to a halt if you don't have enough ram available.<p>Putting a dedicated Varnish server in front of the search servers helped a lot. Using a cdn may also be a viable option, but haven't tried it myself.
That's why I like to use Heroku/EC-2 for launching new webservice. If shits hit the fan, you can jack up the processing power/database/RAM/whatever to scale to your demand. Once you have a good idea of the traffic it generates, you can then move it to a cheaper service.<p>Obviously, it's easy to say that when you're on the bench. Congratulations on the launch by the way.
Once memory goes to swap you already lost. Personally I rarely configure swap on servers, save the DB. I would reconfigure your services to not grow past physical free memory. After that you are going to have to scale servers horizontally.
lamesauce.<p>1. HN should let you pay them $10 and let them hammer your server(s) before your story goes live. good for you. good for them.<p>2. there's a deal at lowendbox right now for a 2GB VPS for $30 a YEAR. you could have a healthy server farm for pretty cheap.
Is there a way to run simulated traffic to determine how your server will react based upon heavier load to try and determine how many people it can serve?
Your project, Cucumbertown, is a cooking/recipe site/platform/network. Hacker News is not your audience/customer. Any "launch" on Hacker News is a fail, regardless of downtime.