[g14i]

Totally agree. In one of our projects (around 200K Python LoC) we had 90% CodCov, but we frequently found some bugs.

Recently we started to heavily use mutation testing and fuzzing to find edge cases on parts of the code that were already "covered" according to the CodCov report. It was definitely worth it.

I highly recommend investing in mutation and fuzzy testing.

[whymarrh]

I was just about to bring up mutation testing. I've had some pretty great success with PIT [1] when writing Java. Code coverage and mutation test coverage pair wonderfully together.

  [1]:https://pitest.org

[TeaVMFan]

Mutation coverage definitely improves the testing process. It is much harder to accidentally write bad tests once you are using a tool like PIT: https://frequal.com/java/HighQualityTestsWithPitMutationTest...

[g14i]

whymarrh, great that you mentioned PIT. This is exactly what we use for Java as well.

For Python we use mutmut [1].

[1] - https://pypi.org/project/mutmut/

[boxed]

Mutmut author here. Great to hear you're using mutmut!

I think it's pretty important that we who know about MT and have used it keep talking about it. It's a pretty great tool that more people should have in their tool belt.

[ay]

test coverage is calculated as % of the locations seen by profiler during the test runs, right ?

This metric completely ignores the fact that every independent conditional doubles the space of the behaviors of the program.

Assuming we have three conditions and three sequential groups of basic blocks chosen corresponding to true (X) and false (~X) values of these conditions, testing A+B+C and ~A+~B+~C code paths gives 100% code coverage but it covers at the very best only 25% of the code behaviors.

[dalke]

There are many types of test coverage measurement. I believe path coverage ("Has every possible route through a given part of the code been executed?") is your latter example. Quoting https://en.wikipedia.org/wiki/Code_coverage which also says:

> Full path coverage, of the type described above, is usually impractical or impossible. ... a general-purpose algorithm for identifying infeasible paths has been proven to be impossible (such an algorithm could be used to solve the halting problem).[14] Basis path testing is for instance a method of achieving complete branch coverage without achieving complete path coverage."