[jere]

[Edit: this comment turned out not to really be about Eve and more about the difficulty of real time MMOs]

This blows my mind. I've been thinking about writing a small multiplayer game for a few weeks. And today I was actually doing back of the napkin calculations. The thing that quickly became clear is bandwidth is O(n^2) where n is the number of players in the same location, since you have to share the location and velocity of each player with each other player.

Here's an example of a calculation I was using. It's the monthly bandwidth in TB of having a certain amount of players in a shared space (3000 in this case), assuming that the location/orientation/velocity of each player's ship is contained in a measly 48 bytes and you attempt 30fps.

(48 * 30 * 3000 * 3000 * 3600 * 24 * 30) / 1e12 = 33592 TB = 33 PB

THIRTY THREE PETA BYTES to simulate a single battle for a month.

I'm looking around at various VPS providers and the bandwidth they offer. Using the naive calculation above and the cheapest Linode offering and it'd take $3 million/month to support a single one of these ongoing battles.

Now, I'm sure the numbers above aren't used in reality. Obviously, you can reduce the updates per second... but not by much. You can't shave much off how much you send. It at least gives you a sense of the orders of magnitude required to support such a thing. Even contemplating supporting 100-1000 people at once is looking very difficult for me to pull off.

[hythloday]

Hey, I used to be a games networking programmer, and your estimate is pretty good. A few corrections that bring it down to something not entirely unaffordable:

* You can bit-pack your structures a lot more efficiently - let's say 2 bytes for a local player ID, 10 bytes for each of velocity/orientation (quaternion represention and heavily quantizing the theta component), and 10 bytes for the location. Clients are never allowed to determine the canonical physics simulation, so all of the above are really just for display purposes, and can be trimmed down as appopriate - we don't need to worry about desyncing (as I'll explain in a bit)

* 30 packets-per-second is way too high for network play, with a game of EVE's mechanics you could probably get away with something perhaps as low as 1 pps. Intermediate simulation of the player entities is done by a technique called dead reckoning that's linked earlier (though in practice you'd use a slight improvement on it to stop entities leaping around the world).

* Sometimes game mechanics allow you to strongly cut down the number packets you send. For example, in EVE, it might be desirable to not send information about a ship if it's completely occluded by another ship, or to only send information about ships that are in a frontal cone ahead of you. This usually doesn't affect the bandwidth function (though sometimes it does), but you can almost always cut a constant factor of (rule of thumb) 50% off the bill.

So that ends up being 32 * 1 * 3000 * 3000 ~= 250MB/s, which looks about right to me. One thing you didn't account for is that you don't typically allow MMO game clients to connect to each other, but to multiplex everything through a server. So it ends up being twice as large as that - 500MB/s.

Good luck with your game! 100 interacting players in an MMO is a challenging target but not an unreachable one for a single-developer game, and it's definitely a very interesting project to undertake.

[GauntletWizard]

Eve is, canonically, 1 cycle/second; That is, the server ticks the game state once per second. Some modules have finer-grained cycle time than that, but their extra is simply saved up tick to tick.

[EarthLaunch]

I'm a game developer bootstrapping an Erlang-based game server that attempts this.

You don't have to send 48 bytes each frame (you can usually send only delta), and 30/sec is only necessary in a reaction-based FPS type of game. Hosting players in a local area, local meaning they can see each other and interact, is n^2. My calculations a year ago came up with a per-minute cost for the amount of AWS hardware it would require to host 1,000 and 10,000 'local' players. Basically, it wouldn't be affordable to have a huge gathering/battle more than a few times a month. It would be VERY expensive. Plus, each player has to have enough download bandwidth.

This type of thing will definitely happen in the future. Clustering is still in its infancy, and it's picking up steam. And there are smart optimizations to be made, such as sending more frequent data about nearer players. You can also design around it by not making players usually local to each other even though there is a single seamless game world. I'm doing that.

[jiggy2011]

Is AWS feasible for large scale game hosting? I would have thought co-location would be much more economical in this case.

[EarthLaunch]

AWS has lower upfront cost, doesn't require hardware management, and it's easier to provision consistent servers during growth or temporary peaks. Ideal for an unstable clustered system being run by a startup.

I'm mainly talking about EC2, since some other offerings aren't suitable for responsive games. I tried a SimpleDB backend to be hip a couple years back, but it has latency and designed failure rates that are impossible.

[jiggy2011]

I haven't done the math but I'm guessing that with a popular game the costs of hardware and hiring someone to do server management would be wiped out by EC2 costs pretty quickly.

Not sure how spikey game traffic is, but it would seem unlikely that your game is super popular for a day and then drops right off the next as would be the case for many websites.

For example if your game business model is charging some nominal fee per month (say $10) you might find that heavy players can burn through way more than $10 worth of EC2 traffic in a month.

Anyone know if any popular games are hosted on EC2?

[thejosh]

Doesn't Sony Japan or Amiga Japan use EC2 for their Playstation 3 servers?

[jiggy2011]

Not sure, though I think the Sony servers are simply used for matchmaking and authentication services and do not actually host most of the games themselves, that is delegated to consoles.

[gamblor956]

Most RTS games, including Starcraft 2, only update the simulation every 100ms or more. Older RTS games like SC1 and AOE2 used "ticks" of 200-250ms. Essentially, every "tick", player input would be received by the server and distributed back to players after validation for processing or display on the next tick (depending on whether processing is done server-side or client-side). The lag is hidden by visual or auditory cues acknowledging the input (i.e., "Yes, sir!" or "For the Horde!").

This greatly reduces the bandwidth requirements because you have extra time to transmit the data to the user; you only send some of the unit data in each packet. Moreover, you can limit the data to units within X radius of the player/camera and simulate any ships outside of this radius, transmitting only major events like the destruction of a ship. (By simulate, I mean the game could just create random ships and explosions client-side to give the effect of a major battle without actually transmitting or receiving data about those ships; they would be eye-candy only.)

[halfasleep]

There's some pretty detailed information about the Unreal Engine [1] that might give you some ideas to think about around network protocols for games. At the time they were dealing with 28.8k modems, so there are quite a few tricks in there about saving bandwidth and prioritising some actions over others. [1] http://udn.epicgames.com/Three/NetworkingOverview.html

[gnarbarian]

I think one thing Eve has going for it (and this is only an assumption) is that it wouldn't need to sync each player's position on each frame. It could reduce the load dramatically by only sending the acceleration / orientation data for a ship.

So if a ship has set a course and has stopped accelerating it doesn't need to send anything else unless that course is changed.

[stingraycharles]

This is exactly how these types of problems are solved. Local clients run simulations based on known locations/directions, and are synced with the central server at a specified interval.

This allows a client to have a light-weight "cached" view of the reality, and the only trade-off is the occasional discrepancy from the reality. However, if you send updates about changes in speed/directions in real time, you get a pretty reliable representation of the state known by the server.

[trout]

Depending on the game (it's been 10 years for me) one 'hack' was to head in one direction, pull the network connection for 3-4 seconds, head in the opposite direction, and plug back in. This allowed you to escape or hide in the terrain fairly reliably.

With higher bandwidths and server capacities I'm guessing these timeouts have been reduced, but never underestimate the player's ability to abuse your trade-offs.

[DigitalJack]

There was an exploit in eve sort of like this. What you did was start warping toward a planet on the far side of the solar system and then kill the application as you got close to it.

The effect was to allow you to sling shot past the destination at warp speed, and if you logged back in at teh right time (seconds later) you could setup a waypoint far outside the outermost celestial body.

(Normally you can only warp to planets and moons, etc... you can't just pick an arbitrary direction and engage warp).

[gizmo686]

That problem is not because of the trade off. The problem is that the server believe the player's client when it provided an updated location. The 'correct' behavior in this case would be that the server resyncs the client, and the player observe's his ship jump to where the server thinks it should be.

[jere]

Sure. It's commonly called dead reckoning as I understand it: http://en.wikipedia.org/wiki/Dead_reckoning#Dead_reckoning_f...

But even if you only send a "real" update once a second, I still don't see how this thing scales (that still leaves 1 PB in the above calculation). And if you send it less than that, I imagine things would start to look rather jittery. The "occasional discrepancy form reality" would be awful.

[stingraycharles]

It depends. If you have 3000 clients and a full sync of all visible ships occurs every second, you get 3000 * 48 = approx 150kb/sec/client. Yes, that means that the server must be able to handle 3000 times that speed, but that still leaves us at about 0.5gbit. Yes, it's a very high load, but then again, these types of situations are extraordinary.

Last but not least, compression is your friend.

[Cthulhu_]

And in Eve, this is even more feasible because, unlike an FPS or even MMO's like WoW, movement is generally slow and controlled. You don't have direct control over your ship; instead, you can only issue commands like 'head for this waypoint', 'orbit this point of interest', etcetera. This would, I think, greatly simplify server communication of movement and such, compared to direct user input where they could change their orientation much more suddenly.

[jere]

I haven't really played Eve, so I can't say. I'm imagining ships flying around and changing course constantly in a battle, but if someone can give a better description, I'd appreciate it. It'd be further reduced by several players sharing a ship... no idea how common that is.

What I'm describing is definitely leaning more towards players that are constantly moving around though. It helps explain to me why in Asheron's Call, for instance, when you had too many people in the same city a "portal storm" arose and people started getting teleported out of the city randomly.

[mbell]

Eve doesn't have WASD controls, there really isn't constant input.

The ship movement command in Eve are limited to:

* Set course / speed (double clicking in space, speed set via clicking a speed dial thing). This is not something that gets constant tweaks, more a 'move in a general direction' command.

* Orbit x @ y distance

* Keep x @ y distance

* Approach x

In addition other actions in eve are relatively long lived, there isn't fps style aiming but 'locking on' and activating modules which have cycles times of 2-60 seconds.

In practice, even in the heat of battle, I doubt the average player gets even close to 1 input action per second over the course of a fight.

[jiggy2011]

Combat in Eve is more D&D and less Wing commander. When I played (a few years ago) you would get text based messages pop up saying stuff like "X hits Y for 38 damage"

[izend]

Eve's gameplay could be represented as a text based game, it is not a fast paced space shooting game.

[gnarbarian]

I also have been toying around with trying to create a MMO fast action space simulation of my own creation. I have a few ideas on how to limit areas of dense players with game mechanics. I haven't even really started on multiplayer beyond the ability to download modify and upload solar systems though.

I think for some really hard engineering problems like this the best course could be to try and find a 'good enough' approach that will allow a significant ammount of players to interact while limiting the upper bound with a game mechanic to make it feel less artificial.

http://code.google.com/p/stableorbit

here are a couple videos from the never ending pyopencl port.

http://www.youtube.com/watch?v=lnOmy1ly6M0 http://www.youtube.com/watch?v=XCvRBHtPbzE

[afterburner]

If you check the video you'll see it's very slow paced and the ships basically don't seem to move at all (maybe tiny unseen ones do I don't know)

http://www.pcgamer.com/2013/01/28/eve-online-battle-asakai/

[gknoy]

Part of the reason it is so slow is that (as mentioned in the article) the server slows time down (as far as 10% of normal) in order to ensure that nothing gets lost.

[dclowd9901]

My guess is they only sync location on transaction (when you fire upon/are fired upon).

[Cthulhu_]

And even that can be optimized away - after all, in Eve, you basically issue commands like 'lock onto this ship' and 'activate this cannon', which will then fire every X seconds - all each client has to know is the target, ship information, and a toggle command for a given weapon, the rest can then be determined completely on the client-side when it comes to the visuals. Every X ticks, data (like health) of the target ship can be sent through - although I reckon that too would be optimized away and only sent on-demand, i.e. when the user actually selects the ship under attack.

[baby]

Can't you do a P2P system when a lot of users are interconnected and then send some information to the prime server?

[rorrr]

The server controls everything. Clients just display whatever the server tells them to (plus maybe some temporal interpolation).

If you start doing P2P, you're opening yourself to all kinds of hacks, and increasing the amount of duplicate data traveling around.

So if you're in a battle of 3000 people, you would have to somehow connect to 3000 people, send 3000 times more data. This would probably saturate your uplink and get you banned by your ISP as a suspected botnet victim.