[hansvm]

Derailing the conversation a bit, what other strategies beyond mutation testing do you use for validating your tests? I've caught test bugs with a few techniques, but none of them are comprehensive, and I'd love to hear more thoughts. Here are a few examples:

(1) Validate assumptions in your tests -- if you think you've initialized a non-empty collection and the test being correct depends on it being non-empty, then add another assert to check that property (within reason; nobody needs to check that a new int[5] has length 0).

(2) Write tests in pairs to test the testing logic -- if your test works by verifying that the results for some optimized code match those from a simple oracle, verify that they don't match those from a broken oracle.

(3) If you're testing that some property holds, find multiple semantically different ways to test it. If the tests in a given group don't agree then at least one of the tests is broken.

[divbzero]

Approaches I take to reduce errors in tests include:

– Keeping tests as short, simple, and few in number as possible. Focus on integration tests for overall input/output, and limit unit tests to the most critical components only.

– Getting extra eyes on the tests. In some cases, this can include product managers who help to define the expected behavior.

– Writing code without continually running the tests. This deviates from strict TDD but allows for a second independent effort at specifying the desired functionality. There are times when writing the code correctly reveals that the tests were written wrong.

[mrkeen]

(1) Randomise your test inputs. If you think that will lead to flaky tests, then you already have flaky code.

(2) Be able to run the software locally in some meaningful way, (as opposed to only seeing the software in action in prod.) My most productive code/test/debug cycle always involves main(). I spend less time imagining what the software will do, and more time seeing what the software actually does.