[dfabulich]

In my experience, no, they don't write tests first, and in fact AAA game companies write almost no automated tests at all.

Instead, AAA teams have large staffs of cheap manual testers--high school and college kids who think they're getting paid to play video games. The managers in charge of these QA teams work their manual testers to the bone, encouraging them to work late and do unpaid overtime. These testers inevitably burn out, and but there's always a new batch of kids coming in to replace them.

When manual QA is that cheap, it turns the economics of automated testing on its head. Why maintain an automated regression suite when you have humans who will do the testing for you?

There's a similar effect on the dev side. A lot of inexperienced developers are willing to take a significant pay cut to work at a AAA game company, just for the dream of working on a AAA game. The devs, too, are notoriously managed as an expendable human resource. "You'd better keep working overtime, or I'll just replace you with another kid who'll work harder than you for less money."

As a result, AAA game developers burn out pretty quickly, too, leaving the game industry for other software companies that both pay better and offer a better work/life balance.

As a result, you'll find that a lot of AAA teams are relatively inexperienced. Sure, there's a core of long-time devs on every team, technical leads who can point the way, but it's not an environment where you can get solid mentoring on the best practices of software development.

Even if one team does pick up TDD, who knows if the same team will even be there in another year or two? Good software practices are regularly "rediscovered" on AAA dev teams, and then forgotten, as the team turns over and their tribal knowledge fades away.

[zamalek]

This is my impression too. There's also good reasoning: TDD is poorly suited to the industry.

For example, how would you test your renderer? Do you run the pipeline and check it against a final image? No, strict TDD would require that the asserted image be hand-crafted. Do you check that each part of the pipeline works as a unit? You can't without running shaders on the GPU (requiring the previous image test).

What game developers are slowly starting to master (already far more than anyone else) is integration testing. Instead of requiring detailed repro steps, just record the testing session in extreme detail and replay it - if anything goes wrong this can be turned into an automated test case.

There are no golden hammers and that includes TDD. TDD is an amazing solution that can provide value in 99.99% of scenarios. We should be scrutinizing areas where it doesn't add value; there is interesting stuff going on there.

[discreteevent]

In my opinion it's the integration tests that provide 99.99% of the value in 99.99% of the cases. Maybe not full integration tests (which can be expensive to run) but at least subsystem tests. These are the ones that find the real bugs. A very high percentage of unit tests once passing tend to continue to pass and only fail when you intentionally change the implementation. i.e. they impede your progress. Low level unit tests are very good for testing an algorithm or something like a parser, say, but that's not a high percentage of the code on the systems that I have worked on.

[andybak]

That matches my intuition. The espousal of unit testing vs other types of tests as "the one true way" actually impeded my adoption of testing in general as I couldn't wrap my head around how to approach it without suffering a huge productivity hit. I thought it was just me "not getting it" but I sense the pendulum is beginning to reverse direction.

[StavrosK]

I agree, this matches my experience. I find that many bugs can crop up in the interface, which is something that TDD inherently doesn't catch, whereas integration testing will both catch bugs in the interface and the units themselves.

[ajeet_dhaliwal]

Exactly my experience too. I have been an automation developer in three teams across two games companies - integration tests provide great value if run continuously and daily, they are especially helpful for uncovering regression. Also the parent describing the sad state of affairs in the AAA games industry is fairly accurate and that leaves little hope for TDD/BDD and other similar practices. One of the sources of all of these problems are business model challenges at a company level which then leads to low pay and high stress creating high attrition. Things just aren't planned out the way you may think they would be. The toss away tradition means that unlike companies developing long running services where automated tests written once may be valuable for even a DECADE, that's not true in games so the ROI isn't quite as high, although this may be changing slightly at least when it comes to engine churn and so in does appear to be slowing down.

Aside - having worked on automated testing in the AAA games industry I have found that good reporting helped managers and other devs better see the ROI from the work put in and I have been involved in creating a service for reporting automated test results across dev teams, see my profile if you're interested and give it a try, you can use it for free and I'd love to get your feedback - send me an email.

[quadcore]

I agree with that.

[Rannath]

You don't test your renderer. The renderer is third party tech in 99% of games. You test that the right calls are being made to the renderer. For that you use a mock. Your test are there to assert correctness in YOUR code. if something exists outside your code you test it another way.

Example: Sam is writing an engine. He has OpenGL, DX9, and DX11 bindings. He also has a Mock that logs calls and checks to see that they match known good logs. Sam then sets up a demo to cycle through all the different graphical capabilities of his engine, this also gives Sam the ability to test his AI/scene crafting code.

[bitwize]

> For example, how would you test your renderer?

Oh, that's easy.

    npm install --save-dev mock-opengl

[bjpbakker]

Mock testing your rendering only assets that you call the methods you expect in your tests. It does not assert rendering in any way.

[shubhamjain]

I find it hard to wrap my head around how the whole AAA VG ecosystem works. How is it able to function? AAA Video Games are highly complex engineering projects but, as I understand, under the hood you have naive and inexperienced kids, horrible engineering practices, and a chaotic work environment. How does this industry manage to hire so many quality engineers and treat them as dispensable that easily?

[user5994461]

Because games are cool and there is no shortage of good developers.

All games are throwaway projects. Write one, sell for a year, gone. There is no maintenance period.

[flohofwoe]

Not true for MMOs and F2P games, these often run for many years some with biweekly or monthly content updates.

[Moru]

Some of the simple but popular mmorpg f2p are licenced engines that you can customise into your own game and just run an update now and then.

Content updates don't need programmers.

[samplonius]

Except for DLC. A AAA title will need to ship at least 4 x DLCs over a 12 to 18 month period. And no doubt the DLC development will expose serious bugs along the way, so the DLC will have to contain patches as well.

The lifespan of AAA titles is longer now that most are purchased digitally.

[EnderMB]

A friend of mine has worked on AAA games in the past, and although he says it's fairly hard work, there is always work out there.

Also, he likes to chop and change every now and then, and the norm seems to be that once a company delivers a game the vast majority of the development team is made redundant. He takes the redundancy, moves to another company/game, and repeats this cycle.

[cbanek]

This echos my experience completely.

One other thing to realize is that you're building a game, you're not building tech. The tech supports the game. A buggy game can be fixed, but a game that works but isn't fun is hard to mend.

A lot of the game is also in the UI, which is notoriously hard to test, let alone use TDD.

[teknologist]

> When manual QA is that cheap, it turns the economics of automated testing on its head. Why maintain an automated regression suite when you have humans who will do the testing for you?

But in practice those who aren't paid a lot don't particularly care much about thorough testing. It requires a systematic approach that a kid out of college who just wants to get paid playing video games isn't going to develop

[cbanek]

They aren't just sitting there playing the games, or designing test cases (usually) - there are actual documents laying out what the test case is and what the manual tester should do to exercise the case. These cases are split up usually among a room of people who check them off one by one for a given build under test.

[microcolonel]

In addition to the steady supply of college-age fools who think that life as a video game tester would be fulfilling; it is also borderline impossible to test many things in video games adequately without human intervention.

Aside from critical technical elements like netplay protocol and file parsing, the only reliable signal you'll get from automated testing is whether the code crashes or not under normal circumstances on the test platform with the tested driver and operating system patch release, most properties of the game are subjective or emergent from the code itself.

[stickfigure]

This is true at a blackbox level but you can pretty effectively create automated tests of finer grained software components. The idea is presumably that if you have a bunch of components that work well, the overall system will tend to work better.

I think this is generally true although it has limits; the higher up you go in the stack, the more value the test has. Still, if it's prohibitively difficult to test higher up the stack, might as well test what you can, no?

I found from my years in at EA in the early 2000s that no matter what the technical issues, the absence of automated testing was at the very least a significant cultural issue. Engineers just didn't think about unit tests. Sounds like that hasn't changed.