Always enjoy these articles. I do think that the simulation strategy deserves a bit more discussion, and the pitfalls of replays are glossed over slightly.<p>Simulating players gives you the benefit of end-to-end testing much more easily (for example, it tests your login flow and disconnect flow, both of which are often written without much care for performance). It also practically demands that you write a scripting console for your game, which is an invaluable resource for error testing, debugging and development both by your engine and content developers. It's true that human players are more unpredictable than bots, but that's not a drawback, because you want to stress-test the hottest part of your game. For these purposes, you can write a small test level (think an arena) and have bots spawn at either end and run towards each other, attacking until they die. This isn't at all representative of how players will play the game but it will very definitely catch performance regressions.<p>Replays depend on two factors, deterministic playback and unchanging content. Deterministic playback is possible (though extremely difficult) to achieve, but it really has to be a priority from the beginning, and it has to have its own testing suite and fixing regressions has to be a priority. It's impossible to retro-fit it onto an existing game (sometimes by design, as non-determinism occasionally is mistaken for randomness). Unchanging content is, obviously, unlikely to happen during development and is much more likely to be possible during betas, but during those time periods you have much more time to devote programmer and content developer time and attention to ancillary features like expanding bot programming.
Great read.<p>I disagree with him on bots, though. CCP has a highly evolved bot system which is tied into their build system, so that unit tests can include player behavior. This isn't directly relevant to load testing (although CCP does that with bots as well), but it certainly demonstrates why bots are worth the effort.<p>And if you've got a full-fledged QA department, bot writing is a good QA engineer task. This is also a decent way to give ambitious QA guys a coding task that doesn't have the potential to affect customer-facing game play.<p>Without violating NDAs I can say I've worked at MMO companies that found bots invaluable for load testing. Replays will definitely catch things that bots won't, but if you're thinking about how many players you can fit in a zone, etc., etc. bots are great.
A very interesting read, but I'm missing one step:<p>Game recording is established as a favorable way to make games better, to avoid a crappy beta experience or problems right after launch.<p>How are you going to get decent recordings in the first place? Playing with a couple developers seems a lousy way to jump start a good selection of games, is it? How is this chicken and egg problem solved? I miss a way to bootstrap the process, load a good set of (varied, representative) replays?
I was confused about the part where he mentions a rack full of 1Us was generating too much heat and the solution was to put less servers in a rack. I can't think of a situation where there is nothing at fault besides the servers. The chillers should be able to keep up with 1Us like that, and it's likely there was something else going on.<p>The situation we're running into at my job is with double density servers (i.e. Dell C6105's). The problem we're having with heating double density servers. Maybe somebody with more experience than me can explain why cooling 1Us would ever be a legitimate problem?