How the EVE Online Servers Deal with a 3,000 Person Battle

The report of the battle they link to (<a href="http://www.pcgamer.com/2013/01/28/eve-online-battle-asakai/" rel="nofollow">http://www.pcgamer.com/2013/01/28/eve-online-battle-asakai/</a>) is even more fascinating to me. You spend hours and hours of effort in order to get a ship, and one way of thinking about them is as the time spent to earn them compressed into 3d form. Unlike (most) other MMO's, battles actually destroy things. So in that youtube clip, you're watching years and years of effort getting evaporated because of a mis-click.<p>In most of the games I play, the pay off of the grinding is getting to enjoy fighting more. But I could never play EVE, because battles would be so anxiety provoking. There would be no pleasure at all in earning a Titan, because I would be forever terrified of losing it.<p>Of course, all games are kind of there to destroy your time. But for some reason it doesn't feel quite as wasted when you have a virtual thing there that is bad-ass in proportion to your time investment.

[Edit: this comment turned out not to really be about Eve and more about the difficulty of real time MMOs]<p>This blows my mind. I've been thinking about writing a small multiplayer game for a few weeks. And today I was actually doing back of the napkin calculations. The thing that quickly became clear is bandwidth is O(n^2) where n is the number of players in the same location, since you have to share the location and velocity of each player with each other player.<p>Here's an example of a calculation I was using. It's the monthly bandwidth in TB of having a certain amount of players in a shared space (3000 in this case), assuming that the location/orientation/velocity of each player's ship is contained in a measly 48 bytes and you attempt 30fps.<p>(48 * 30 * 3000 * 3000 * 3600 * 24 * 30) / 1e12 = 33592 TB = 33 PB<p>THIRTY THREE PETA BYTES to simulate a single battle for a month.<p>I'm looking around at various VPS providers and the bandwidth they offer. Using the naive calculation above and the cheapest Linode offering and it'd take $3 million/month to support a single one of these ongoing battles.<p>Now, I'm sure the numbers above aren't used in reality. Obviously, you can reduce the updates per second... but not by much. You can't shave much off how much you send. It at least gives you a sense of the orders of magnitude required to support such a thing. Even contemplating supporting 100-1000 people at once is looking very difficult for me to pull off.

Fun facts: This fight was between the Honeybadger Coalition (HBC) and the Clusterfuck Coalition (CFC). The core of the Honeybadger Coalition is TEST Alliance, and the core of TEST Alliance is Dreddit, the "original" Reddit-based corp (there are / have been several, some of which predate Dreddit and have since folded).<p>HBC routed the CFC, which has Goonswarm as its core, which is organized around the Something Awful forums.<p>TEST Alliance was nurtured by the CFC for a long time, splitting off only in the past year to forge a more independent identity. As such, the hostilities between these old friends is hard to gauge. At this point, the presumption is that it's all in good fun. But that may change.

It's like each solar system is a VM, and they can't move their VMs to a new physical server without disconnecting all clients. And all of their physical servers are at 100% load all of the time? Oh, I guess it's 100% utilization, not 100% load. As in, they don't spin down servers to save power during off-hours I guess.<p>The time dilation is a neat solution to the server load problem, but it's sooo annoying as a player. In beta, it was interesting to watch the entire game desync and grind to a halt, but we could still chat and look around. Interesting, but frustrating.

This is a nice article.<p>A friend of mine did some research for DARPA/NSF on internet 'crowds.' His research was looking at the question of fractal gatherings, which was basically you were 'near' the people around you in the virtual 3D space and could hear what they were saying, you were adjacent to people who were one space away but could hear them if they 'shouted' and the crowd was at a gathering that was sharing an experience from a presenter who was 'projected'. Basically a virtual concert venue for example where process affinity scheduling took into account where you "were" in the 3D space. The questions are many and as far as I can tell the folks who research this are few and far between.<p>Process migration continues to be a hard thing to do. VMware and others have made progress, virtualized interfaces and peripherals, self contained 'state'. Works 'ok' at the VM level but still doesn't work at all AFAIK at the 'thread' level.<p>World of Warcraft addresses this somewhat by 'instances' which is that under certain circumstances (entering a dungeon for example) you and your party of 5, 10, 25, or 40 people all transition (through a loading screen) into a place with nobody else (except you). They can dedicate machines to host an instance and do process migration at that level.<p>DARPA is interested of course because they would love to have a way of connecting warfighters into a virtual command and control center regardless of where they are physically. Essentially being in two places at once is a big force multiplier.

This was particularly interesting for me. I've already built a scalable MMO server architecture in Erlang, which hosts all players on one world, using server nodes that can be moved between machines. I calculate that my server architecture can handle up to 10,000 moving players in a local area. That's characters, not EVE-style ships (characters are heavier).<p>If anyone is interested in talking about this, helping, or funding me, please get in touch. My server is called Proxima and my game is First Earth (you can find it if you're interested). I've been slowly bootstrapping it for 3 years.

I always wondered how Eve coped with what seems like a massively data heavy game, keeping track of battles and logging data especially for epic battles where alliances are pitting their expensive ships against each other. Pretty insightful article, would love to know a bit more about their development processes and how they go about pushing new changes up. It's such a sensitive game, changing just one thing could throw the whole game into disarray. It's crazy when you think that they not only have to track battles, but they also have to capture every other action and movement within the game; mining, agent missions, the marketplace and all of the other components that rely on accurate and up-to-date data capture and realtime reporting.<p>What makes this all even more crazier is the game has been around for over 10 years and counting. I still play Eve everyday, once you get past the menial tasks, you can have some real fun with this game being a drug runner, miner or just straight up renegade roaming the infinite environment.

The environment EVE creates in almost every aspect of the game is just fascinating.<p>It is the only game I ever played I actually felt an enormous thrill and adrenaline rush when playing pvp. It doesn't matter if you're pvping alone or with a fleet. The fact that you actually lose/gain on a fight differentiates the game from any other MMO. Events that you may have experienced in-game can be recorded forever and actually impacts the whole game universe.<p>I really hope more companies invest into having one persistent world in their MMOs. Instead we see this segregated servers in which great players/achievements/events are limited to this little server. It ends up lessening the interest of many players, specially older ones.<p>Funny thing is that, normally, when the hit MMO stops being a hit they end up merging lots of servers anyways. Which IMHO end up lessening even more the time people have spent on that game.

Second Life (I may be wrong on the game itself) talked about how it manages game scale across many servers.<p>The entire game is a virtual machine.<p>When a player leaves a server boundary, their player VM state is packaged up and sent to the next world server and then resumed on that server.<p>Players don't even notice they crossed server boundaries, and this was possible because the game can be paused mid flight for that player - pretty neat.

I haven't played EVE in years, but I always love reading these reports to reminisce. It took a huge time investment but it was a lot of fun. Reading that CFC apparently has hundreds of titans and lost 3 just in this battle is crazy. I was playing when the first Titan was built, and I remember distinctly when the first one was destroyed.

On moving the game to a beefier server: <i>“This is the machine that systems get reinforced on when players request that. Unfortunately, the same thing above applies – anyone in the system when the move happens gets disconnected. Because of this, it’s basically never done,”</i><p>It seems like a fairly small addition to the game logic to allow the game to freeze for a few seconds and be forcibly moved to another server. Is this kind of scenario so rare that it's not worth the trouble, or am I missing something else?

And of course it's a video of the former Goonswarm[GOONS] now [GEWNS]

Aww, and now I want to start playing EVE again :).<p>Great article, great battle.

This alliance has some guy who does simultaneous translation to russian?

tl/dr: Sharding and slowing everything down.