[jaaron]

You're misunderstanding what the parent poster was saying: when working on a large AAA game, content management (art, game design, etc.) is as much a bottleneck as engineering efforts. And a number of those bugs you're concerned about are rooted in content too, not lower level engine bugs.

I've worked about half of my career in the game industry. I've practiced TDD and written automated tests (and frameworks) for desktop, web and mobile apps. Some of those have been in the medical industry where the testing is crucial. I say this to make it clear that I'm familiar with solid software engineering practices.

With that in mind, games are the hardest software I've encountered for writing automated tests. It's just notoriously difficult to do in an effective manner. It's not impossible, but it's incredibly difficult.

[reader_mode]

>I've worked about half of my career in the game industry. I've practiced TDD and written automated tests (and frameworks) for desktop, web and mobile apps. Some of those have been in the medical industry where the testing is crucial. I say this to make it clear that I'm familiar with solid software engineering practices.

Did you work in the game industry before the other stuff or after ? Because I went that way game dev -> application development - and frankly a lot of the SW engineering practices from game dev were terrible in the transition - not because I couldn't apply them in game dev - but because I didn't know about them - and nobody around me told me - and I haven't seen it in code from others.

>With that in mind, games are the hardest software I've encountered for writing automated tests. It's just notoriously difficult to do in an effective manner. It's not impossible, but it's incredibly difficult.

There's a ton of low hanging fruit - running recorded controllers, partial scenario tests, gold copy rendering tests, smoke tests, regression testing - frankly it's not that hard to raise the bar from 0. I'm not up to date in the industry so maybe they aren't at 0 anymore but from occasionally keeping tabs and playing games occasionally I would say it hasn't moved far.

Just the number of regressions in MMOs for example where you could easily code tests for the stuff that was fixed is an obvious example that nobody was doing regression testing or adding tests after fixes. And this is for MMOs that have an incentive to keep a healthy codebase (not just ship and forget)

[SeanBoocock]

Everything you have outlined was standard practice at studios I have worked at for more than five years. The lower in the stack you go, the easier it is to do things like unit tests. In my experience the section of the game industry that is the easiest to test in an automated way is AAA mobile. Where I work in “HD” AAA, it is considerably harder to test in the same ways but we do where it is effective. Don’t mistake the failures of some games or studios as a valid indictment of the industry.

[WorldMaker]

> Don’t mistake the failures of some games or studios as a valid indictment of the industry.

It's an industry focused intentionally on as minimal as possible code and best practice sharing for fear of talent poaching/project poaching. It's an industry where the "best practice" is still reinvent as many wheels as possible every other game and open source little to nothing. The worst mistakes of the worst games/studios should likely remain a valid indictment of the industry as a whole when the industry itself is so focused on making sure the tide rises as few boats as possible.

[marta_weber]

It's easy if the game engine you are using would display an in-memory model with deterministic algorithms. I have only briefly worked in the game industry and it's been more than enough for me :D. There were like 4-5 people with "10 years+" experience working on a Candy Crush clone. It was literally riddled with bugs and they used Unity's physics engine to make the stones fall (I mean, it doesn't get any more ridiculous than this).

So I came in, throw it all away and built within 2 days a deterministic, fully tested and testable stone engine that would implement all the effects management wanted to see, some of which would have been close to impossible to implement with Unity's physics engine. The idea is basically to use the smallest time unit you want to support, and then base all algorithms on that. You don't work on milli-seconds, because that is for the engine to fill out. You work on seconds, for CandyCrush at least. Each time step is basically a second. The time in-between is filled out by potentially non-deteministic animations and particle effects and whatnot. But every second, the whole scenery will synchronize with the deterministic engine that drives it all.

While I didn't have the chance to try this on shooters or MMORPGs, I think especially for MMORPGs, this would be a perfect fit. It solves sooo many problems, I would probably need a day listing all the benefits. Are there drawbacks? Probably, I can't think of one right now, except that it goes against anything normal game developers believe in.

I think at some point, the gaming industry forked away from solid software engineering practises.

[oppositelock]

It's a lot more difficult than you make it out to be.

I'm a former game engine lead from an Activision studio. The whole engine architecture was my responsibility, testing included.

You can easily write tests for all the deterministic stuff; math, physics, save games, utility code, filesystem code, etc.

However, where it gets difficult is testing the content. Every platform has some quirks which force you to create art custom made for that platform (your other choice is to use the minimal subset of what they all support). If you want a good looking game, your content has several versions, so now, things like intersection and pathing code are slightly different between platforms. GPU behavior is different enough that using GPU's for anything results in multiple test code paths. AI isn't always fully deterministic, by design, making it even more difficult to test. So, what you have in the end are some deterministic tests to make sure your foundation is sound, and "fuzzy" tests to catch many problems in the higher level constructs, and your final line of defense are play testers, which is an awful job!

Now, we engineers know how to make the games stable and reliable, and our time estimates are frequently at odds with the PM's. You MUST make Christmas, which in the days that I worked, where CD's must be mastered, meant code freeze and asset freeze in October sometime. So, in this mad scramble to make a hard deadline, quality suffered.

It was much more important in the old days (PS1, PS2, GameCube, N64, etc), to get it right the first time, as you couldn't ship patches, but the notion of patching games allowed release versions to be more buggy, since people kicked the can down the road. Granted, the earlier consoles only supported simpler games, and the diffiulty was dealing with the particulars of the system, not the game itself. The PS1 had no Z-Buffer, the PS2 had a CPU/GPU combo built by a crazy person with an PS1 as its IO controller, and the PS3 required you to write DMA management code, while the GameCube had a memory model that was nuts and matrices were only 4x3 so you couldn't do projections. So, you spent most of your time dealing with these quirks and the game ended up simpler due to the effort being spent elsewhere.

Now, it's much easier, consoles are effectively PC's with powerful general purpose CPU's and GPU's, so it should be possible to write high quality, reusable engines.

[411111111111111]

I've always wondered why big studios don't employ some people that just write bots to play through scenarios etc.

I mean the biggest issue with creating good bots is getting the required data to make decisions such as "you're getting attacked, you need to use this ability" etc, but if you're actually the company writing the game, there shouldn't be any issues creating APIs for that. Or even going directly into memory and read it out, you don't need to worry about getting banned after all.

Imagine Integrationtests in the form of bot scenarios. Is that too much overhead to implement in the usually very budged oriented setting of game development?

[nradov]

You can write a bot player that follows a script or applies some simple decision rules. The hard part is detecting whether or not the game is operating correctly at each step. It's not like a simple web application where things happen in discrete steps and you can inspect the DOM to verify that it contains the right nodes.

[dahart]

Lots of big and small studios have tried to automate game testing. They are interested in it, so if it seems easy and fun you should do it, I’m sure there are jobs out there writing game testing bots.

That said, it would be wise to assume that if you have not heard much about this and wondered why, the answer is probably that it’s harder than you think.

Having been a game lead for a decade, I can safely say that getting game telemetry into the test bots is not the hardest part of the job, it’s one of the easiest things to do. One example of something much more difficult would be testing the game while it changes. Don’t forget that writing a cheat bot for a game that’s done is nothing like testing games that are in development and changing every day.

[kaba0]

How would it tell that something is wrong? Like, it went through a wall, and the app let it do so - what’s the problem?

Testing non-deterministic code paths are a really hard problem because you don’t really know what to test for.