[fwlr]

Arguments against testing tend to fall prey to the von Neumann Objection: they insist there is something tests can’t catch, and then they tell you precisely what it is that tests can’t catch… so you can always imagine writing tests for that specific thing.

E.g. this article uses an example of removing the number 5, causing the developer to have to implement a base-9 numbering system. Unit tests that confirm this custom base number system is working as expected would be extremely reassuring to have. Alternatively, you could keep the base-10 system everyone is familiar with, and just have logic to eliminate or transform any 5s. This would normally be far too risky, but high coverage testing could provide strong enough assurance to trust that your “patched base-10” isn’t letting any 5s through.

The same is true for the other examples - unit testing feels like the first thing I’d reach for when told about flaming numbers.

[HideousKojima]

Nah, my objection to unit testing is that too often it devolves into what I call "Testing that the code does what the code does." If you find yourself often writing code that also requires updating or rewriting unit tests, your tests are mostly worthless. Unit tests are best for when you have a predefined spec, or you have encountered a specific bug previously and make a test to ensure it doesn't reoccur, or you want to make sure certain weird edge cases are handled correctly. But the obsession with things like 100% unit test coverage is a counterproductive waste of time.

[fwlr]

I partially agree - I would say more specifically “those situations are the easiest to write good tests for”, ie having a predefined spec will strongly guide you towards writing good and useful tests.

“Testing that the code does what it does” is of course a terrible waste of both the time spent writing those tests, and of future time spent writing code under those tests. With skill and practice at writing tests, you make that mistake less often. Perhaps there’s a bit of a self-fulfilling prophecy for game developers: due to industry convention, they’re unfamiliar with writing tests, they try writing tests, they end up with a superfluous-yet-restrictive test suite, thus proving the wisdom of the industry convention against testing.

[renox]

Unfortunately this can also be the case of integration test : I spent the last week trying to understand if the regressio n in an integration test was a bug or if it was the previous behaviour which was buggy..

[jayd16]

The lesson is more about the degree of churn and how game rules are not hard rules. A valid base 9 number system is NOT a design goal and doing that work can be a waste.

It's like testing that the website landing page is blue. Sure you can but breaking that rule is certainly valid and you'll end up ripping out a lot of tests that way.

Now, instead of calcifying the designer's whims, testing should be focused around things that actually need to make sense, ie abstract systems, data structures etc etc.

[fwlr]

Tests that “calcify the designer’s whims” - great way to put it - can be quite useful if your job description happens to be “carrying out the whims of the designer” (and for many of us, it is!)

With high coverage and dry-ish tests, changing the tests first and seeing which files start failing can function as a substitute for find+replace - by altering the tests to reflect the whims, it’ll tell you all the places you need to change your code to express said whims.

[magoghm]

Tests can't catch race conditions in multithreaded code. Now that I told you what the tests can't catch, can you imagine writing tests for that specific thing?

[a_t48]

I've written tests around multithreaded code, but they typically catch them in a statistical manner - either running a bit of code many times over to try and catch an edge condition, or by overloading the system to persuade rarer orderings to occur.

There's also https://clang.llvm.org/docs/ThreadSafetyAnalysis.html which can statically catch some threading issues, though I've not used it much myself.

[jonex]

tsan will catch a bunch of potential race conditions for you, under the condition that you run it somehow. How to make sure it's run? Well, add a test for the relevant code and add it to your tsan run in your CI and you'll certainly catch a bunch of race conditions over time.

This has saved me a bunch of times when I've be doing work in code with proneness to those kind of issues. Sometimes it will just lead to a flaky test, but the investigation of the flake will usually find the root cause in the end.

[mistercow]

I’ve written tests to do exactly that, by adding carefully placed locks that allow the test to control the pace at which each thread advances. It’s not fun but you can do it.

[magoghm]

Doesn't inserting locks affect the memory hierarchy consistency mechanisms and therefore interfere with possible race conditions?

[mistercow]

That’s not a situation I’ve encountered but “race condition” is an extremely broad category.

[distortionfield]

> Tests can't catch race conditions in multithreaded code.

Citation needed.

> can you imagine

Yes I can, because several languages have tooling built specifically for finding those race conditions.

If you built it, you can test it. If you can’t test it, you don’t understand what you built.