... and then the customer called and asked why the graphs on the front page were wrong even though they clearly just edited hugetable.<p>You explain them that it'll just take a little while to be updated, but the customer didn't like that answer. The data needs to always be current.<p>Apparently, you need to flush parts of the cache as new data arrives. Unfortunately though, you can't as memcache is a strict key/value store. So you change how you name the cache keys and make them dependent of, say the max(timestamp) of your hugetable.<p>Load goes back up to 2 because all requests now still have to check the table.<p>But it's still not as bad.<p>Until the next phone call...
For some reason, I started reading the story with the assumption that it was a "don't do it this way" tutorial, and I got very nervous towards the end. ("But that's exactly how I use memcache!")
Programmer and Sysadmin were either very lucky, or not working on anything important, or else they would have been fired or gone out of business. You can't just add caching and magically expect things to work. You have to think hard about expiration policies and test to make sure you aren't going to get wrong answers, or else you need to prove that wrong answers are ok.