Hacker News new | ask | show | jobs
by Emjayen 1246 days ago
As someone who fixes up old games in my spare time, and Age of Empires II being one of them, I'll provide a bit of trivia about the game's internals:

- The AI system is not part of the deterministic simulation. This was surprising to me, and after contacting one of the original programmers it was explained that it was due to a desynchronization bug that the "AI and network programmers weren't able to fix it in time".

A consequence of this design regression is that, due to the AI now being authoritatively run by the designated host player, network congestion issues arose which lead to a clear series of progressively more aggressive optimizations to reduce egress traffic. This primarily consisted of a very simple filter (mentioned in the article) which dropped duplicate commands in the common submission path, meaning it applied to both local user and AI commands, along with batching of AI-submitted commands which would be flushed at rather arbitrary times.

I'll note that I've restored AI being deterministic in my project.

- A rather obscure determinism bug resulted from their compiler's implementation of a few CRT routines, namely fsin/fcos and a few others, which leveraged the specialized ISA instructions of the same name. The problem being that these transcendental functions are beyond the scope of the IEEE 754 spec. and ergo are hardware implementation-dependent. In practice, contemporary Intel/AMD chip families produce bitwise the same result, however those around the time of AoE are known to diverge on results to some small margin (as confirmed by an Intel engineer on a thread I came across while researching).

- The game employs a dirty-update system for rendering, not only that it's at a scanline granularity. This was something I was very pleased to see, as it's a exceedingly rare to see such an important optimization in games of this era (although common in earlier eras)

- There's some "interesting" naming conventions, one being prefixing member variables names with "value" -- there's even a "valueValue". Very little consistency in general in this regard, a reflection of independence between teams working on different components.

- While there are attempts at validation of input into the simulation (albeit woefully inadequate), it relies on ad hoc inclusion of PID (player ID) fields within commands. This is entirely useless, as this information is not authoritative and controlled by the players, permitting them to "spoof" the contextual information required for validation.

This is one of the more perplexing aspects of the engine, especially given the necessary information about the origin of a command is ofcourse available.

(As this information was later publicly published by other individuals I don't see a problem with elaborating on it here as I have)

- An example of missing the wood for the trees: session information goes through an ad hoc compression for its wire form (just bitpacks fields) to conserve bandwidth, however the architectural choice is to synchronize this session state by just having the host broadcast the state --dirty or not-- every 200ms, flooding the network pointlessly (atleast in the 18.8k days)

- While Age of Empires is somewhat notorious for its poor multiplayer performance (notably when contrasted with say, the recent "Definitive Edition"), and while its implementation of this synchronization model is certainly rather juvenile, it should be understood that the final MP gameplay issues are primarily due to the choice of a peer-to-peer topology over the public internet, which at the time was the most reasonable.

While superficially P2P may seem like it should achieve the lowest latencies for instance, the reality is that the primary determinant is the characteristics --jitter, delays and packet-loss, reordering, ect-- of the path between two hosts and a P2P architecture means there's n*n paths (network egress/ingress paths are typically asymmetric). In contrast, a server/client model you not only have far fewer routes, but datacenters are located at critical points in the network, roughly analogous to comparing travelling A->B along a freeway versus via the maze of residential streets.

- The state checksum "algorithms" involved are bordering on useless, atleast for state-tracing (such as when debugging desync. bugs.). They appear to have been devised by way of believing doing "a bunch of random bitwise ops" constitutes sufficient mixing -- a quick test demonstrated that csum collisions were not just possible, but occured sometimes for over 80% of inputs.

--

There's a lot more that could be said but I feel that's enough for now.

6 comments

Sending state updates every 200ms made my spidey sense tingle. Most industrial control systems of the 90s would work on continuous 200ms polling through the network. Mostly RS485 links at 9600 baud.

The thing is that this is perfectly reasonable if your network infrastructure is known. If you have fixed bandwidth and deterministic packet sizes, then you can do math and know what the behavior will be. Determinism is good! Also this assumes the network is single purpose. Which it was! For games in the 90s it was! This isn’t bad design, it was good design for the network infrastructure people would have had at the time!

Certainly true for local and/or controlled networks, however over the public internet where you're competing for bandwidth with many others over limited (and sometimes already congested-) links it's a rather questionable choice.
Absolute left field question. Have you touched the other RTS contemporary of the time, Red Alert 2 in any detail?

I've been told the source code for it leaked some long time ago and has floated around for the past two decades or so. While most people contend that EA had lost the source code to the Command & Conquer series games many years ago when the studio was closed down and assets were shipped to "DICE" in Stockholm

RTS-wise it's just been Blizzard games otherwise; Starcraft and Warcraft III, the latter of which I did the most work with, mainly multiplayer optimizations and writing a server implementation.

However, I have been entertaining the idea of supporting other deterministic (RTS-) games, adapting them the same way --rearchitecting the multiplayer system/code-- as I have done for Age of Empires II, providing a unified platform for these games. The first candidate that came to mind was the Red Alert series.

I remember using some kind of resource explorer, looking into the scripts powering the AIs. If I remember correctly, it didn't get "smarter" at higher levels, it just cheated and gave itself more resources every minute, lol.
I've heard this repeated quite a bit, and while I haven't checked the AI scripts themselves (although I don't recall/imagine there being a script facility to give oneself resources..) I can say that, engine-wise, the only handicap behavior for AI that I've noticed is:

- On the highest difficulty ("Hardest"), AIs are bestowed 500 to all resources on next-age research completion. - Work rates for researches are skewed to be slower for the lowest-difficulty setting.

> I'll note that I've restored AI being deterministic in my project.

Are you the OpenAge author, or maybe Voobly? I've never read such detailed info on AoE's internals online. Thanks for the post.

> In practice, contemporary Intel/AMD chip families produce bitwise the same result, however those around the time of AoE are known to diverge on results to some small margin

That little bit of validation I needed for why I avoid AMD CPUs. (not entirely serious)

No, I'm quite the outsider to the community having only been involved in the game for a few years (besides as a teenager when it was released) and working alone.

The project in question --a platform for AoE2 multiplayer /w general game improvements-- is currently under development, although the core game/MP has been functional for over a year now and our little beta-testing group plays regularly.

> There's some "interesting" naming conventions, one being prefixing member variables names with "value" -- there's even a "valueValue". Very little consistency in general in this regard, a reflection of independence between teams working on different components.

Actually suffixing! i.e. objectsValue.

Good catch, Morten :)
How are you able to work on the game? Are you employed to do so? Do you reverse engineer? Or has the source leaked?
Reverse engineering (which is not actually something I particularly enjoy doing frankly)