I tried it. If you had a super serious algorithm where you were will to spend any money to test it (a central security algorithm for example), it might be worth it.
The problem I had is that there were many many mutations which led to code which was functionally identical, just a bit slower, or took a different route. You have to manually verify every mutation to check if this.
For one mutation it took me 30 minutes to convince myself the algorithm was still doing the same thing, just in a different way.
I haven't worked with it extensively, but from what I've seen, it's far beyond what most teams should consider. It's basically a more thorough method of measuring test coverage, pointing out cases that have not been covered by your tests yet. However, the number of tests that it would have you write to reach 100% mutation coverage is not justified by the number of bugs you'll catch, unless the impact of any bug is very high (I suppose at NASA?). In fact, the amount of work required to even check which tests you missed is already not justified.
I usually use coverage as a tool to remind me of pieces of code I intended but forgot to test [1], so setting it to an arbitrary percentage is not that useful, in my opinion. If 100% is not feasible usually as well, that makes Mutation Testing in general not worth the effort.
I think mutation testing really shines in code bases that are already heavily tested, because it let's you discover test cases that you don't actually need. Tests are a burden since you have to adapt them when you change the behavior. With mutation testing you can prune your test code by identifying tests that test very similar behavior.
Would that be done based on which (different) tests killed a particular mutant?
Often an integration test would catch the multiple mutations also being caught by (different) unit tests.
I assume that you mean that if a certain (broader) test kills the same mutants as X unit tests, those X tests are not really necessary?
I've looked into https://github.com/boxed/mutmut and https://github.com/sixty-north/cosmic-ray for Python project, and there it is only important that a mutant gets killed, but not how often and by which tests (therefore you can use `pytest -x` to continue with the next mutation after the first test failed due to it).
Me neither, I liked the concept of mutation testing. (I was using this performing changes manually, without knowing this technique has a nice name).
I would appreciate if somebody points out a mutation framework /tools for .net
There's something that used to be called Pex, and now is apparently called IntelliTest [1], but I believe it's only available in the Enterprise edition of Visual Studio.
The problem I had is that there were many many mutations which led to code which was functionally identical, just a bit slower, or took a different route. You have to manually verify every mutation to check if this.
For one mutation it took me 30 minutes to convince myself the algorithm was still doing the same thing, just in a different way.