[existencebox]

Let me take this opportunity to say "you guys do some damn good work". Certainly one of the more impressive games I play, stability/longevity/feature growth wise, especially considering the size of GGG.

Topical question; however, since the thread on a whole has focused on the more visible but more finniky-to-test aspects of these games. Do your same statements apply more or less to the backend/networking infra? I remember the decent ball of work you guys wrote about a while back when introducing the lockstep change, was curious what went on behind the scenes to support and validate that from an ops/maintainability perspective, if you can speak on that at all.

[Negitivefrags]

For backend we test with a load testing bot, but we don't do it every commit, just after making larger changes to the backend. I would like to test more, and I'm actually working on better automation for load testing at the moment so that it's easier for us to run them.

Testing lockstep was quite an endeavour. The approach I used was to add a very verbose log to the client and the server which logs basically every gameplay relevant decision. That includes moving a single unit, updating a stat, etc.

Then every time a line is logged, we update a hash, and then send the hash to the client. The client, being in lockstep, should be able to come up with the same hash. If the hash is different, then the client and server both break and we can then analyse the state and see what is different.

As we discovered issues, we added more and more log lines and fixed bugs until everything ran perfectly the same on the client and server.