[bvrmn]

> Coverage measurements are too easy to “game” — you can get to 100% coverage without meaningfully testing all or even most of your code

Still it's a good low bar for testing. It's easy and rises code quality. I have very good results with coverage driving colleagues to write tests. And on code review we can discuss how to make tests more useful and robust and how to decrease number of mocks, etc.

[tasuki]

Hard disagree: 100% coverage is not a "good low bar" and does not increase code quality.

Depending on the language and the particular project, my sweet spot for test coverage is between 30-70%, testing the tricky bits.

I've seen 100% code coverage with tests for all the getters and setters. These tests were not only 100% useless, they actively hindered any changes to the system.

[Lutger]

This is true.

You can have bad unittests which make the system worse and you would be better of without them. You can also have useless unittests with 100% coverage, which is pretty much the same as bad tests because more code means more bugs and more work. Unittests are also code after all.

The only thing you can say about a very low coverage is that you probably don't have good tests. That's not a very useful metric, since you likely already know that.

The metric 'coverage' is almost useless. Code coverage starts to be useful once you let go of it as a goal and ignore the total percentage number. I found it is very useful though if you can generate detailed reports on each line of code or better yet, each branch in the code, indicating whether that line or branch is tested. Eyeball all the lines which don't have tests and ask yourself: would it be useful to add a test exercising this codepath? How do I make sure it works and what cases can I think of that could go wrong? This doesn't automatically lead to good tests, but it helps you spot where you should focus your testing efforts.

Code coverage is a good tool to help think of test cases, as a metric for the total codebase it is nearly useless.

[hbrn]

> Code coverage starts to be useful once you let go of it as a goal and ignore the total percentage number

When a measure becomes a target, it ceases to be a good measure.

It takes immense discipline to actually let go of a metric to keep it valuable.

[bvrmn]

Funny thing that 100% coverage really helps for dynamic typed languages. It's easy and helpful.

[bvrmn]

> I've seen 100% code coverage with tests for all the getters and setters. These tests were not only 100% useless, they actively hindered any changes to the system.

It's a red flag to blame high coverage for fragile tests. Use narrow public component interfaces to reach code parts and you simultaneously gain robust tests which can be used during refactoring and you can be guided by coverage to generate test cases. Bob Martin has a great article: https://blog.cleancoder.com/uncle-bob/2017/10/03/TestContrav...

[BerislavLopac]

One useful technique for checking whether the tests are actually meaningful is mutation testing - mutmut is a great Python implementation: https://mutmut.readthedocs.io

[hbrn]

> It's easy and rises code quality

Absolutely not. This leads to testing being invasive and driving the design of your software, usually at the cost of something else (like readability). Testing is a tool, you can't let it turn into a goal.

[bvrmn]

Could you elaborate with python-oriented examples? I tend to agree for static typing languages like Java — to fully test you have to go a total DI path. It leads to bloat and additional layers. I don't see anything similar for python because you have to do nothing to bring your code into a test environment.

> Testing is a tool, you can't let it turn into a goal.

Yep, and I use testing as a tool to be sure we ship quality code. It's 2x important for our case, we don't have control on hosts where our product is run and 100% coverage was a salvation. We even start to ship new versions without any manual QA.

[hbrn]

My examples are just personal anecdotes, you can dismiss them by saying "our team won't make the same mistake".

If your goal is 100% coverage then it will turn testing into ritual and only give you the illusion of quality. Instead of testing inputs and edge cases, you will focus on testing lines of code.

There's a good illustration of uselessness of 100% coverage in one of Raymond Hettinger talks: https://www.youtube.com/watch?v=ARKbfWk4Xyw

> I use testing as a tool to be sure we ship quality code

I suspect we have different definitions of quality, and your might include testing, so I doubt I will be able to convince you.