Ask HN: Is TDD or BDD used in AAA gaming industry?

[jcfausto]

TDD, BDD, or any other form or development driven by tests first are very popular approaches in web development or even non-web developments.

I'm curious to know if these approaches are also considered a MUST for AAA game development.

Is there anyone that has worked with these approaches while developing games or know any big AAA game company that uses test first approaches as something required during the development?

[dfabulich]

In my experience, no, they don't write tests first, and in fact AAA game companies write almost no automated tests at all.

Instead, AAA teams have large staffs of cheap manual testers--high school and college kids who think they're getting paid to play video games. The managers in charge of these QA teams work their manual testers to the bone, encouraging them to work late and do unpaid overtime. These testers inevitably burn out, and but there's always a new batch of kids coming in to replace them.

When manual QA is that cheap, it turns the economics of automated testing on its head. Why maintain an automated regression suite when you have humans who will do the testing for you?

There's a similar effect on the dev side. A lot of inexperienced developers are willing to take a significant pay cut to work at a AAA game company, just for the dream of working on a AAA game. The devs, too, are notoriously managed as an expendable human resource. "You'd better keep working overtime, or I'll just replace you with another kid who'll work harder than you for less money."

As a result, AAA game developers burn out pretty quickly, too, leaving the game industry for other software companies that both pay better and offer a better work/life balance.

As a result, you'll find that a lot of AAA teams are relatively inexperienced. Sure, there's a core of long-time devs on every team, technical leads who can point the way, but it's not an environment where you can get solid mentoring on the best practices of software development.

Even if one team does pick up TDD, who knows if the same team will even be there in another year or two? Good software practices are regularly "rediscovered" on AAA dev teams, and then forgotten, as the team turns over and their tribal knowledge fades away.

[zamalek]

This is my impression too. There's also good reasoning: TDD is poorly suited to the industry.

For example, how would you test your renderer? Do you run the pipeline and check it against a final image? No, strict TDD would require that the asserted image be hand-crafted. Do you check that each part of the pipeline works as a unit? You can't without running shaders on the GPU (requiring the previous image test).

What game developers are slowly starting to master (already far more than anyone else) is integration testing. Instead of requiring detailed repro steps, just record the testing session in extreme detail and replay it - if anything goes wrong this can be turned into an automated test case.

There are no golden hammers and that includes TDD. TDD is an amazing solution that can provide value in 99.99% of scenarios. We should be scrutinizing areas where it doesn't add value; there is interesting stuff going on there.

[discreteevent]

In my opinion it's the integration tests that provide 99.99% of the value in 99.99% of the cases. Maybe not full integration tests (which can be expensive to run) but at least subsystem tests. These are the ones that find the real bugs. A very high percentage of unit tests once passing tend to continue to pass and only fail when you intentionally change the implementation. i.e. they impede your progress. Low level unit tests are very good for testing an algorithm or something like a parser, say, but that's not a high percentage of the code on the systems that I have worked on.

[andybak]

That matches my intuition. The espousal of unit testing vs other types of tests as "the one true way" actually impeded my adoption of testing in general as I couldn't wrap my head around how to approach it without suffering a huge productivity hit. I thought it was just me "not getting it" but I sense the pendulum is beginning to reverse direction.

[StavrosK]

I agree, this matches my experience. I find that many bugs can crop up in the interface, which is something that TDD inherently doesn't catch, whereas integration testing will both catch bugs in the interface and the units themselves.

[ajeet_dhaliwal]

Exactly my experience too. I have been an automation developer in three teams across two games companies - integration tests provide great value if run continuously and daily, they are especially helpful for uncovering regression. Also the parent describing the sad state of affairs in the AAA games industry is fairly accurate and that leaves little hope for TDD/BDD and other similar practices. One of the sources of all of these problems are business model challenges at a company level which then leads to low pay and high stress creating high attrition. Things just aren't planned out the way you may think they would be. The toss away tradition means that unlike companies developing long running services where automated tests written once may be valuable for even a DECADE, that's not true in games so the ROI isn't quite as high, although this may be changing slightly at least when it comes to engine churn and so in does appear to be slowing down.

Aside - having worked on automated testing in the AAA games industry I have found that good reporting helped managers and other devs better see the ROI from the work put in and I have been involved in creating a service for reporting automated test results across dev teams, see my profile if you're interested and give it a try, you can use it for free and I'd love to get your feedback - send me an email.

[quadcore]

I agree with that.

[Rannath]

You don't test your renderer. The renderer is third party tech in 99% of games. You test that the right calls are being made to the renderer. For that you use a mock. Your test are there to assert correctness in YOUR code. if something exists outside your code you test it another way.

Example: Sam is writing an engine. He has OpenGL, DX9, and DX11 bindings. He also has a Mock that logs calls and checks to see that they match known good logs. Sam then sets up a demo to cycle through all the different graphical capabilities of his engine, this also gives Sam the ability to test his AI/scene crafting code.

[bitwize]

> For example, how would you test your renderer?

Oh, that's easy.

    npm install --save-dev mock-opengl

[bjpbakker]

Mock testing your rendering only assets that you call the methods you expect in your tests. It does not assert rendering in any way.

[shubhamjain]

I find it hard to wrap my head around how the whole AAA VG ecosystem works. How is it able to function? AAA Video Games are highly complex engineering projects but, as I understand, under the hood you have naive and inexperienced kids, horrible engineering practices, and a chaotic work environment. How does this industry manage to hire so many quality engineers and treat them as dispensable that easily?

[user5994461]

Because games are cool and there is no shortage of good developers.

All games are throwaway projects. Write one, sell for a year, gone. There is no maintenance period.

[flohofwoe]

Not true for MMOs and F2P games, these often run for many years some with biweekly or monthly content updates.

[Moru]

Some of the simple but popular mmorpg f2p are licenced engines that you can customise into your own game and just run an update now and then.

Content updates don't need programmers.

[samplonius]

Except for DLC. A AAA title will need to ship at least 4 x DLCs over a 12 to 18 month period. And no doubt the DLC development will expose serious bugs along the way, so the DLC will have to contain patches as well.

The lifespan of AAA titles is longer now that most are purchased digitally.

[EnderMB]

A friend of mine has worked on AAA games in the past, and although he says it's fairly hard work, there is always work out there.

Also, he likes to chop and change every now and then, and the norm seems to be that once a company delivers a game the vast majority of the development team is made redundant. He takes the redundancy, moves to another company/game, and repeats this cycle.

[cbanek]

This echos my experience completely.

One other thing to realize is that you're building a game, you're not building tech. The tech supports the game. A buggy game can be fixed, but a game that works but isn't fun is hard to mend.

A lot of the game is also in the UI, which is notoriously hard to test, let alone use TDD.

[teknologist]

> When manual QA is that cheap, it turns the economics of automated testing on its head. Why maintain an automated regression suite when you have humans who will do the testing for you?

But in practice those who aren't paid a lot don't particularly care much about thorough testing. It requires a systematic approach that a kid out of college who just wants to get paid playing video games isn't going to develop

[cbanek]

They aren't just sitting there playing the games, or designing test cases (usually) - there are actual documents laying out what the test case is and what the manual tester should do to exercise the case. These cases are split up usually among a room of people who check them off one by one for a given build under test.

[microcolonel]

In addition to the steady supply of college-age fools who think that life as a video game tester would be fulfilling; it is also borderline impossible to test many things in video games adequately without human intervention.

Aside from critical technical elements like netplay protocol and file parsing, the only reliable signal you'll get from automated testing is whether the code crashes or not under normal circumstances on the test platform with the tested driver and operating system patch release, most properties of the game are subjective or emergent from the code itself.

[stickfigure]

This is true at a blackbox level but you can pretty effectively create automated tests of finer grained software components. The idea is presumably that if you have a bunch of components that work well, the overall system will tend to work better.

I think this is generally true although it has limits; the higher up you go in the stack, the more value the test has. Still, if it's prohibitively difficult to test higher up the stack, might as well test what you can, no?

I found from my years in at EA in the early 2000s that no matter what the technical issues, the absence of automated testing was at the very least a significant cultural issue. Engineers just didn't think about unit tests. Sounds like that hasn't changed.

[speps]

If you want to experience that in the AAA game industry, I work at Rare Ltd (UK, part of Microsoft Studios). We've been doing TDD and Continuous Integration for about 5-6 years now starting with Kinect Sports Rivals. For Sea of Thieves[0] (our current project), we've extended that to Continuous Delivery and it's been just a complete game changer in regards to overtime and crunch. For example, the onboarding exercise is basically a TDD kata that has to use our coding guidelines and build farm so you get to know how we work without having to dive right away into Unreal Engine 4. We use TeamCity for our CI solution.

And yes, we are recruiting : https://www.rare.co.uk/careers

[0]: https://www.seaofthieves.com

[SCdF]

Can you give some examples of what you're actually testing? Perhaps an example of a specific test statement, the kind of level the tests sit at etc, what gets covered and what doesn't etc?

Are you testing the rendering engine? Lighting / physics / shaders / collision detect?

Are you testing scripting? Win conditions? AI (probably not for SoT I guess)?

Are you testing the core engine? File mgmt? memory mgmt? input reading etc?

As someone who is a developer and a gamer but has never combined the two, the idea of TDD in that scenario is fascinating.

[speps]

We are unit testing all new code that can be unit tested, trying to do dependency injection when possible so mocking is easier.

We are testing the rendering engine, lighting, shaders, etc. in a visual test framework we created ourselves that compares each run of the tests with the previous and uses SSIM to output a comparison metric. At the moment, that's still checked by someone every week but could be automated. However, when changing rendering code or shaders, we do require that a test result is shown during code review.

For physics and collision detection, we do have tests covering that too, a lot of them involving our FFT water simulation as well.

We are testing the whole flow of the game from start menu to entering a session, matchmaking, loading a map etc. I think AI also has some good coverage, notably for the navmesh which is essential.

The core engine is Unreal Engine 4. It's not an engine made for testing unfortunately, even though the vanilla version has some (but not nearly enough). We've stuck with a specific version for a while now as upgrading usually meant that we'd also get new bugs from the new version. All the core is covered by all the existing tests, we've added tests where we thought it was critical and where we fixed or improved internal code of the engine.

Before joining Rare, I really wasn't aware of any testing whatsoever in the video games industry. It's been an eye opener and I wouldn't do it any other way.

Reading other comments, we might not follow TDD all the time or to the letter but the point is that testing itself is a very part of everything we do as engineers. That allows our QA team to focus on stuff that is actually broken and can't be tested (such as if a feature is actually fun).

[SCdF]

Thanks for your reply.

> We are testing the rendering engine, lighting, shaders, etc. in a visual test framework we created ourselves that compares each run of the tests with the previous and uses SSIM to output a comparison metric

So would this be something like rendering a frame of a test scene (utah teapot? real in-game scene?) in various states of rendering (with / without lighting, with / without specular / normals etc), and then making sure they look the same each time?

> Before joining Rare, I really wasn't aware of any testing whatsoever in the video games industry. It's been an eye opener and I wouldn't do it any other way.

You should definitely consider a GDC/etc talk, I imagine it would be an eye opener for a lot of people.

[speps]

> So would this be something like rendering a frame of a test scene (utah teapot? real in-game scene?) in various states of rendering (with / without lighting, with / without specular / normals etc), and then making sure they look the same each time?

Yes, exactly, it's small test scenes we setup ourselves whenever we add a new rendering feature or fix a rendering bug. Some of them are still not deterministic as we're using Compute Shaders for some stuff which is hard to make deterministic.

> You should definitely consider a GDC/etc talk, I imagine it would be an eye opener for a lot of people.

We are currently looking for topics. We'll see what comes out of that discussion internally :)

[peruvian]

Why did you guys make a new Banjo Kazooie game in 2008 but made it about Lego cars? :-)

[Negitivefrags]

I work on a game called Path of Exile.

At our studio we don't do TDD, but we do continuous integration with a lot of asset testing.

In the early days, we tried doing unit tests, but honestly, it's very hard in a game. We had little success testing gameplay code.

What we have a huge amount of is asset tests.

We have 4 times more non-programmers than programmers working on the game, so most of what is created isn't code anyway.

Every time anyone commits anything, it goes through our continuous integration pipeline, and any asset that changed, or is dependent on an asset that changed, is loaded and tested. This includes test loading entire levels.

Test loading every asset catches most types of failures.

We try, where possible, to add specific tests for common mistakes in order to catch things as early as possible.

For example, if an animation has "walk" or "run" in the name, then we test it also contains footstep events in the animation somewhere. If a model is used as a monster we test that it has an attachment point called "frame_upper" onto which various effects can be attached. We look for things like bounding box of an object being far away from the co-ordinate origin (a common mistake when making a big maya scene with lots of exported objects in it).

We also do a test start of the servers, which includes loading all of their data, and a test start of the client.

All of this means that a build that goes through green is unlikely to cause crashes for other people trying to work on the game.

We do, of course, also have an excellent QA team.

[existencebox]

Let me take this opportunity to say "you guys do some damn good work". Certainly one of the more impressive games I play, stability/longevity/feature growth wise, especially considering the size of GGG.

Topical question; however, since the thread on a whole has focused on the more visible but more finniky-to-test aspects of these games. Do your same statements apply more or less to the backend/networking infra? I remember the decent ball of work you guys wrote about a while back when introducing the lockstep change, was curious what went on behind the scenes to support and validate that from an ops/maintainability perspective, if you can speak on that at all.

[Negitivefrags]

For backend we test with a load testing bot, but we don't do it every commit, just after making larger changes to the backend. I would like to test more, and I'm actually working on better automation for load testing at the moment so that it's easier for us to run them.

Testing lockstep was quite an endeavour. The approach I used was to add a very verbose log to the client and the server which logs basically every gameplay relevant decision. That includes moving a single unit, updating a stat, etc.

Then every time a line is logged, we update a hash, and then send the hash to the client. The client, being in lockstep, should be able to come up with the same hash. If the hash is different, then the client and server both break and we can then analyse the state and see what is different.

As we discovered issues, we added more and more log lines and fixed bugs until everything ran perfectly the same on the client and server.

[skibz]

Was the difficulty of unit testing game code due to the complexity and unusual nature of PoE's gameplay? (Specifically, the whole spell/attack/support gem system and various other item modifiers)

[EngineerBetter]

As a one-time indie game developer who went to work at a 250-person games studio, my answer based on limited experience is "probably not".

BDD in particular is tricky, because beyond the level of abstraction of the engine, you don't really know what you're building. How do you express that a thing should be fun, or weapons reloads feel punchy, in a BDD spec? How do you automatically test that?

Games code exists purely to elicit reactions in human brains. We don't have the technology yet to examine the desired state change in the target system.

Regular apps have the same problem, but with less expensive assets and less iteration. "As a user, I should feel that the login transition is slick" is a spec I can imagine that is UX-related. However, we can probably all agree on what slick is, and failing that you could user-test it to prove that the implementation is acceptable. A game is made of many complex adaptive interacting systems, so every change needs subjective validation.

On the subject of complex adaptive systems: many games feature them and so exhibit fundamentally unpredictable emergent behaviour. This can be hard to test for.

The most testing I've done in a games project is to figure what it is you're trying to build, then unit test the implementation when you've worked out if it's fun.

[fuwafuwa]

Short answer: No. Gamedevs have to keep showing testing zealots the door.

Long answer: The problems with the approach are manyfold. There are a few points where testing is automatable, but they don't describe entire game projects well.

* What are you testing when you add gameplay? You are testing for a whole set of design concerns across the project, not just a technical specification or quality of service metric. You cannot do it in isolation and get the results you want at the speed you need, because you need tons of feedback to discover what a complex game is currently strong or weak at. The only reasonable way to gain the necessary feedback is to allow prototyping code to drive near-term changes, manually playtest it, and then factor out the most stable parts of the resulting soup where you can. This works against test-first because you don't really know what you're testing. You throw in a feature on a hunch - "maybe this will give the experience we're looking for" - and see how it behaves hands-on. If it works, there are still usually ramifications and elaborations that didn't surface up-front. And once it does work it's hard to break out into a sandbox because the interesting part of the behavior is in the coupling to other systems and features.

* Games deal with vast quantities of mutable state strapped onto relatively straightforward data pipelines: the parts that are most testable are engine and toolchain core elements like asset builds and rendering systems. These parts do see unit tests and integration tests, although there isn't total consensus on how much testing is needed or how it should be done. They are the most like business and application software, though, since their development can be more structured around technical goals and quality metrics.

* Many bugs are ultimately data bugs. Some designer or artist set up the wrong binding of assets or made a valid yet broken combination of parameters. Games tend to fly apart very suddenly when they operate on bad data. But the code is doing exactly what it should do for that set of inputs - and if it's a failure it's at the level of process and tooling, not the runtime code.

[stuntkite]

I worked for a AAA game company in the game server department and I would say that with the schedule and how the shop was directed it was nearly impossible to do TDD. I can't speak for the engine side, but my guess is that it was not possible over there either.

I bet like anything through, if you go to shops that have strong direction, can work with a small team and over deliver, those standards paradigms are used.

Check out the tests in the Quake3 Source Code. It is a shining example for sure.

https://github.com/id-Software/Quake-III-Arena

[moomin]

My biggest observation with AAA games is that the coding is actually a fairly small part of the work. Most of it is one huge asset pipeline. And I don't think we have particularly good tech for testing what "looks right". Maybe computer vision will get that good eventually, but it's not even a thing right now. The same applies to the gross properties of renderers.

Even in big-spec-up-front triple-A games, there's a lot of exploratory work where you honestly don't know what you want. BDD and TDD don't help there either, rapid prototyping does. (TDD presupposes you know what you want, BDD presupposes someone does and you need to find and talk to them.)

None of which is to say that there aren't many places where test friendly design and regression tests wouldn't help. But the surface isn't the same as the average business app.

[taspeotis]

Hi, this article [1] isn't quite what you're after but you might be interested in it anyway.

[1] https://engineering.riotgames.com/news/automated-testing-lea...

[je42]

Nice !

Looks, like they also automatically create tickets for defects ! !

[wolfgke]

Croteam applied lots of automated testing by bots in The Talos Principle to ensure that changes in level design cannot lead to a situation where the game cannot be completed anymore:

> https://venturebeat.com/2014/11/04/the-talos-principle-under...

> http://www.gdcvault.com/play/1022784/Fast-Iteration-Tools-in...

[sk0g]

That's not TDD, really, but it's a clever way of testing both the AI, and level changes, to a certain extent. If you drop a fucking boulder and you can't cross it yeah, it will light you up with red floodlights, but if a certain corner drops you into the nether...

Unless the AI also knows to check each possible interaction, movement, etc.

[majewsky]

Can you clarify why you do not consider this TDD?

[wolfgke]

I think that if one is meticulous, sk0g is actually right (source: https://en.wikipedia.org/w/index.php?title=Test-driven_devel...): In TDD one begins anything (in particular any new feature) with writing a test for it, then one runs all tests to see if something fails and only then starts to write code if necessary until all tests pass.

[sk0g]

In TDD you only write code if it makes a red light green, so to speak.

First you make the test, then you run it. It will obviously fail, and then you implement code that starts passing tests.

[gleenn]

Having never worked at a gaming company but having a friend who was the head of testing at one very recognizable shop, the answer was definitely no, at least not there. Its definitely a big enough industry that I'm sure some does though.

Also, last time I wrote any 3D code, it at least appeared to be incredibly hard to test that some visual thing was just so. Of course there are still things like business logic that surely could be tested, but it's just not going to be as straight forward as having some code generate some HTML string and then asserting things against it. Those last-mile things you might want to test are always a challenge IMHO.

I learned a lot of my coding chops at a Rails shop that was very in to testing, so that's definitely my perspective, YMMV.

[brad0]

In my experience no. Testing was not even considered. I worked on mobile games 5 years ago so things may have changed.

Most of the time we had strict timelines with feature milestones. We were building mobile apps of licenced titles.

The lack of TDD was replaced with full time QA, sometimes multiple on the one game.

[purrcat259]

Factorio is a good example of a test driven developed game:

https://www.factorio.com/blog/post/fff-186

[Lapsa]

factorio doesn't really qualify as coming from AAA gaming industry. and factorio devs are on another level.

[purrcat259]

Yes, I agree. I forgot the AAA part after reading several comments before I posted mine.

[hacker_9]

Is that TDD? Looks like automated tests run after each commit.

[falcolas]

I agree; automated integration tests. Still incredibly useful (per the devs), and not quite the industry standard.

[ZenoArrow]

I'm not in the game industry, but from what little I know, this is probably going to be a tricky question to answer.

Reason being, a high proportion of AAA game development is done using off-the-shelf game engines. Of course it's still necessary to write code for the game logic and features, but because of the stable base this code is written against, it might not be seen as necessary to follow a strict TDD/BDD approach.

[clemParis]

A high proportion of web development is done using somehow "off-the-shelf" web frameworks, and yet people write unit test/functional tests for the application logic and features. Despite how stable the base this code is written against.

[maccard]

> a high proportion of AAA game development is done using off-the-shelf game engines.

I don't think this is particularly true. A high proportion of game development is, but many AAA studios are using either home-rolled engines, or started with an existing engine (Unreal or iDTech) and what they're currently running looks absolutely nothing like the original engine.

[sk0g]

I agree with both of you, sorta?

The problem is, most of the major game dev companies have their own engines. Unreal Engine is massive in AAA games, but most EA titles use Frostbite, Valve: Source, Ubisoft: Anvil, Crytek: CryEngine, Bethesda: Gamebryo, and the CoD devs use IW, which is a fork of id tech engine.

However, this doesn't mean that they aren't using off-the-shelf engines. As long as you count in-house "shelves".

[stonewhite]

A colleauge working at a AAA Soccer game gave some insight about that a few years back.

Their automated testing was mostly bruteforce stuff like,

* Placing a character at penalty spot and making it shoot with every possible angle and strength, making sure the ball didn't launch to space.

* They would leave a computer vs. computer game running for hours / days to make sure no kind of random crash would happen.

[grogenaut]

I worked on a AAA game. I controlled the networking code (web services really) down and the entire server infra for our project. All of that was full TDD but it was also all greenfield by me who had done heavy TDD before and working with an external team who was all TDD. I even wrote integration tests from a stub client that ran on the playstation as part of the build to hit my server and a stub server to do wire testing for the client as part of my builds. Also had a full load regression test as part of my build that ran every checkin on a smaller slower cluster.

The main game had no tests... except that you need to realize that we had 480GB of asset compiler integration tests and 32GB of game engine integration tests.

As a heavy TDD person it took me a long time to realize that this can be very sufficient though it's a longer turn around loop. The test coverage just from being able to boot the game up is quite large. This is something that many non-game type programs have real issues with, they can't easily just "run all the user data". I know we had one project where we did a lot of TDD but it wasn't actually as good as just running the entire input set every build which I eventually switched us to. This was a non-game data processing system. Doing this + unit testing gave us great coverage and few bugs.

Generally major bugs were found within 15 minutes by another engineer, eg the next person to sync, build or run the engine or art compiler.

I do think we could have unit tested the core collections libraries in the engine to great effect and faster turn around but the build was not really segregated in a way that we would take advantage of this. It was just a monolithic build; the collections objects were heavily header included so there really would have been a build step to built them into their own test exe, run them then build the game. So I'm not sure how much benefit we would have gotten. I think in 4 years I saw one collections issue that lasted past just launching the engine and not having it crash.

[Nysa]

I am not working directly in gamedev but I have some insights.

I've seen growing interest in automated testing from mobile apps.

Basically guys were writing regression testing with framework like Selenium (UI testing) that goes throughout most common scenarios. (I think that was described by huuugegames.com on some presentation)

The main idea was to let QA focus on "exploratory" tests. (Testing rules of the game, design decisions or performance)

TDD & BDD can be used gamedev but not in places you will think of first.

I will simplify it vastly: programmers make tools for designers to use.

Tools are made by programmers - tools can be done with TDD and/or BDD. Games are made by designer - they use tools to make rules, systems etc.

Therefore gameplay is make mostly out of data (designer's parameters set in engine/tools) - you can't test your data with TDD/BDD.

[oscarlgz]

Kind of related: http://www.eurogamer.net/articles/2017-07-19-a-devs-eye-view...

[drawkbox]

Game development projects are usually focused on tech/data/network/engine, asset creation/pipeline, behaviors and gameplay.

In the tech/data/network/engine areas there may be TDD/BDD if the engine is to run multiple titles beyond the current project, if not the budget may not even allow for time. Possibly other areas may have tests in the behavior/gameplay areas: components, AI etc, this depends on the current dev culture usually.

Really the main thing in game development is shipping and building fun, if tests help you do that faster/better then you do them. I think most devs developing anything that is not game specific but tech/data/network/engine related likes to have tests to rely on, time permitting.

Most studios I have worked with that do some TDD/BDD/automated testing it usually happens on the "tech" or core/engine teams rather than the "production" or "live" teams. The tech teams are working on engine/network/data while the live/production teams are using the engine/tools/scripts to make assets and gameplay. The tech team isn't as locked to game launch dates/crunches and has more time to be thorough because most of their code will last across multiple titles.

For the most part, in the asset creation/pipeline, behaviors and gameplay area, more of that is visual and QA based. There are testing companies and departments setup just for the gameplay/feature testing. So much of game development is going for fun and a good mechanic that actually playing it and testing it live is more effective than automated tests in some areas. Although there are some automated test tools in Unity[1], Unreal[2] and custom engines maybe for very rote areas of gameplay (physics engine, collectables, asset loading, very common unchanging actions etc).

Many times on the production/live side of game development, gameplay and behaviors change so much that any automated testing becomes a lag on dev/iteration time, or aren't updated and stagnate as the changes happen so quickly day to day. This is especially true in iterative development during prototyping or pre/post-production, since it is mostly visual the benefits are outweighed by the speed required to ship.

[1] https://unity3d.com/unity/qa/test-tools

[2] https://docs.unrealengine.com/latest/INT/Programming/Automat...

[je42]

> Many times on the production/live side of game development, gameplay and behaviors change so much that any automated testing becomes a lag on dev/iteration time, or aren't updated and stagnate as the changes happen so quickly day to day. This is especially true in iterative development during prototyping or pre/post-production, since it is mostly visual the benefits are outweighed by the speed required to ship.

So you think, you if you do testing you are overall slower ? Skipping tests makes youufaster short term.

Long term, you are slow as hell because of the lack of coverage and manual effort.

Also, especially test help you with large scale changes. Because, these tell you what breaks what shouldn't have been broken.

Further, test is not the same as test.

In order to do efficiently testing you need to apply the test-pyramid.

i.e. have lots of unittest. less integration test. less system tests. less e2e tests.

another guiding principle is to focus automated testing on the things are most curicial for the success of the product, or the most difficult to do manually.

also, you need to chose the tools that support your testing stratgey. if you work against your tooling. oh boy, you have a journey ahead.

(For example unity unittesting framework was pretty crappy last time i check -- 3 years ago )

[drawkbox]

> So you think, you if you do testing you are overall slower ? Skipping tests makes you faster short term.

Not at all long run, but in the prototyping stage it may be a hindrance or when you are going for a fun gameplay mechanic over coverage. When you lock something down then tests make sense in this scenario. You'd always want tests for unit testing data, network, core tech, but might not need it when you are building gameplay that the test is really a user experiencing it/tuning it. Even writing some rendering or gameplay tests it takes time to get them right before they are actually useful tests.

Agree on all the other items there are many levels of testing. Usually core libraries are covered, when it comes to in-progress prototyping/pre-production, it can definitely slow you down (by adding weight to changes) if it is changing constantly and not yet locked down. Another area that weighs into it is how much budget and time that the code might be in use, if it is just a tool or something that isn't needed for live code (something prototyping or to help develop it, i.e. asset packing or the like) it might not be worth it to surround it with unit/integration testing.