[mrsilencedogood]

"serve [seemingly] <50ms latency N:N coordination between thousands of clients, sometimes who are all literally in the same place (events, cities etc) expecting to observe their and other users actions ~immediately"

This still sounds like a really hard problem, I can't imagine trying to do it 20 years ago. (WoW turns 20 in Nov, and it wasn't the first, though perhaps the first to reach such supercilious scale?)

[pjc50]

It only really works because it isn't N:N, it's N:1:N, and it's aggressively sharded (you have to have a separate EU and US shard if you don't want one to have transatlantic latency, which is 50ms all on its own). While you can have a few thousand people in one place, the interactions are limited in ways which avoid N^2 blowups.

You'd be surprised what's possible with efficient in-memory client-server code.

[don_neufeld]

Yup, and there are some non- obvious ways to offload compute that were necessary back in those days.

Half the cluster for Planetside was edge servers whose entire job was to handle inbound packets and distribute them to the relevant other clients. One copy of each update was sent to the zone server, and it was then responsible for updating adjacency lists so the edge could do its job. This gave us much closer to linear rather than quadratic scaling. Even then the packet forwarding code in the edge servers didn’t do full packet decode, it just dug into the header deep enough to determine packet type, which allowed it to know how to forward.

I think we launched on ~20 servers per world, each of which was a dual core 300 MHz P3. That could handle 5000 players in a world. Including raytracing their projectiles against an 8km x 8km mesh ;)

I can’t even find processor benchmarks that far back, but I’d guess we ran the entire cluster on less CPU than most cell phones have today.

[astrobe_]

I've heard of tricks like player position prediction/interpolation by the clients, in particular for the FPS/Sim genres, which I guess is mainly aiming at mitigating "lag spikes" more than optimizing the network traffic, but it looks like it could give some leeway to servers as well ?

[pjc50]

Prediction lets you see through time but doesn't reduce the overall amount of work you have to do - the update confirming the prediction has to be sent eventually.

Also, remember that the server cannot afford to trust the clients about anything, not can it let them have too much information about the world.