[oppositelock]

It's a lot more difficult than you make it out to be.

I'm a former game engine lead from an Activision studio. The whole engine architecture was my responsibility, testing included.

You can easily write tests for all the deterministic stuff; math, physics, save games, utility code, filesystem code, etc.

However, where it gets difficult is testing the content. Every platform has some quirks which force you to create art custom made for that platform (your other choice is to use the minimal subset of what they all support). If you want a good looking game, your content has several versions, so now, things like intersection and pathing code are slightly different between platforms. GPU behavior is different enough that using GPU's for anything results in multiple test code paths. AI isn't always fully deterministic, by design, making it even more difficult to test. So, what you have in the end are some deterministic tests to make sure your foundation is sound, and "fuzzy" tests to catch many problems in the higher level constructs, and your final line of defense are play testers, which is an awful job!

Now, we engineers know how to make the games stable and reliable, and our time estimates are frequently at odds with the PM's. You MUST make Christmas, which in the days that I worked, where CD's must be mastered, meant code freeze and asset freeze in October sometime. So, in this mad scramble to make a hard deadline, quality suffered.

It was much more important in the old days (PS1, PS2, GameCube, N64, etc), to get it right the first time, as you couldn't ship patches, but the notion of patching games allowed release versions to be more buggy, since people kicked the can down the road. Granted, the earlier consoles only supported simpler games, and the diffiulty was dealing with the particulars of the system, not the game itself. The PS1 had no Z-Buffer, the PS2 had a CPU/GPU combo built by a crazy person with an PS1 as its IO controller, and the PS3 required you to write DMA management code, while the GameCube had a memory model that was nuts and matrices were only 4x3 so you couldn't do projections. So, you spent most of your time dealing with these quirks and the game ended up simpler due to the effort being spent elsewhere.

Now, it's much easier, consoles are effectively PC's with powerful general purpose CPU's and GPU's, so it should be possible to write high quality, reusable engines.

[411111111111111]

I've always wondered why big studios don't employ some people that just write bots to play through scenarios etc.

I mean the biggest issue with creating good bots is getting the required data to make decisions such as "you're getting attacked, you need to use this ability" etc, but if you're actually the company writing the game, there shouldn't be any issues creating APIs for that. Or even going directly into memory and read it out, you don't need to worry about getting banned after all.

Imagine Integrationtests in the form of bot scenarios. Is that too much overhead to implement in the usually very budged oriented setting of game development?

[nradov]

You can write a bot player that follows a script or applies some simple decision rules. The hard part is detecting whether or not the game is operating correctly at each step. It's not like a simple web application where things happen in discrete steps and you can inspect the DOM to verify that it contains the right nodes.

[dahart]

Lots of big and small studios have tried to automate game testing. They are interested in it, so if it seems easy and fun you should do it, I’m sure there are jobs out there writing game testing bots.

That said, it would be wise to assume that if you have not heard much about this and wondered why, the answer is probably that it’s harder than you think.

Having been a game lead for a decade, I can safely say that getting game telemetry into the test bots is not the hardest part of the job, it’s one of the easiest things to do. One example of something much more difficult would be testing the game while it changes. Don’t forget that writing a cheat bot for a game that’s done is nothing like testing games that are in development and changing every day.

[kaba0]

How would it tell that something is wrong? Like, it went through a wall, and the app let it do so - what’s the problem?

Testing non-deterministic code paths are a really hard problem because you don’t really know what to test for.