[wilgertvelinga]

The solution to that is mutation tests. They force your tests to actually verify the implementation instead of just running the code to fake coverage. https://en.m.wikipedia.org/wiki/Mutation_testing Tools and frameworks exist for almost all languages. Some examples:

- stryker-mutator (C#, Typescript)

- pitest (Java)

- mutatest (Python)

[misja111]

I have tried those out for a large Java project. The problem is that it is just too slow to use with a large unit test suite. What mutation testing does, is running all the tests in your suite multiple times while changing some of your code base ("mutation points") to see if it affects the test outcome (it should). So say that your test suite normally takes 10 minutes, then a full mutation suite can easily be a factor 50 slower, which means 8 hours.

[wilgertvelinga]

Find ways to reduce those 10 minutes. Some ideas: - make the individual tests run faster - remove obsolete tests - increase parallellism - use a tool that can determine which tests to run based on what has changed, for example nx.dev.

[jononor]

Perfect for a daily CI job?

[polynomial]

"daily CI"?

[rafaelmn]

People tend to use CI as a synonym for automated build pipeline.

[polynomial]

So I've heard

[LadyCailin]

You don’t block on that, you run it on a continuous loop in a CI. Then, if tests fail, you are limited to the checkins over the last day or whatever, and you can more easily bisect to find the offending PR, and file a bug against that dev to fix, or just revert the change. But you dont run the full suite locally, though there are ways for stryker anyways to only test code that changed, which you can run locally, or in the PR build.

[MichaelNolan]

Mutation testing is really amazing for some types of code. And it definitely solves the problem of “fake unit tests”. Though sometimes it does force the tests to be a little too tightly coupled with the code. (Though that usually indicates that the code could be refactored)

[nprateem]

Or instead of adding more rules and tech solutions, discuss with the team and gain buy-in on code quality and processes.

[DylanSp]

The paper in the OP mentions mutation testing at the end when talking about future work and open problems (section 7). They think it'd be very useful, but it's "challenging to deploy such computationally demanding techniques at the scale we would require".

[pawelduda]

Also mutant (ruby). Good but unfortunately with costly subscription for commercial use