[hinkley]

I begin to relax around 85%.

It’s perfectly possible for a project to hit 85% without much untoward going on. Running the code and testing the code are not the same, and as the numbers go up I tend to find shenanigans.

I’d rather have an honest 85% than any dishonest number you could want. My butt starts to pucker again above 95%, because it’s all lies.

There also comes a point where running certain tests by hand is faster or more reliable than having a test for it. False positives (failing tests) are expensive, and in a way that is often counted poorly.

For a start, a false positive doesn’t cost the time of the test, it costs the time of the entire test suite times three - the one you wasted, the one you do over, and time between getting a failure and noticing the failure.

It also costs trust in the tools. Red tests become untrusted, which really hurts when you have a test that has false negatives. People become conditioned to hit the run button again without understanding the error, which then blows up farther down the line, possibly in production.

[netsectoday]

> My butt starts to pucker again above 95%, because it’s all lies.

With the way you are describing false-positives; it sounds like most of your tests are written as integration or feature specs. Exercising the UI at 95% coverage is insane (maybe that is why you said it's all lies). You will definitely be pulling your hair out with false-positives and with a large enough project you'll never get it 100% passing each day.

The answer here is to shift your tests from the UI or integration level down to unit tests. You should test at the function or model level with unit tests to pin all possible execution paths, then perform a sanity check test in the UI. For example; exhaustively test your login module at the unit level, then perform a sanity check at the UI level. Moving exhaustive tests down to the function or unit level increases test speed because you don't have to load or render the UI to perform the tests (which also eliminates most classes of false-positive tests).

If you or your team have issues with false-positives at the unit level then something isn't right with your test harness, OR you need to git-blame to identify the person on your team that needs some mentoring.

[hinkley]

Testing is hard. Most of the people who tell you otherwise are full of... bravado. Sometimes they have the worst tests.

When I started working with more dynamic languages I took comfort in the promises from TDD folks that you can write tests to cover all of these things, but I've seen how tests are written in practice and it's not good. I've met maybe ten people in twenty years who are really competent at writing bulletproof tests (as in, "You wrote tests for that? Enough said" versus Trust but Verify) and I am not at the top of that list. Frequently enough, I find eyerollers that have my name by them in the commit history.

I'm not saying I'm ready to go back to static languages because of this, but I will say that I see evidence for hope on the static analysis side for once in a very long time.

As you say, the problem is often discerning between unit tests and functional tests. Some people clearly can't tell the difference, but a number of people have confessed discomfort at writing unit tests (something about my frank communication style seems to invite people to share their fears with me, and I hear things that the "everything is great" people don't due to self-censorship). That these unit tests seem juvenile, pedestrian. Remedial. But to my mind that's the point. The cognitive load of determining why I just caused a bunch of tests to go red should be near zero, since I'm still trying to keep my head wrapped around the new code I'm in the midst of writing.

For about as long as I've been writing tests I've worked with people who think writing their own mocks or helper functions in tests is a good use of time. They take some work, so once you have them working they write five more tests that use them. The poor SOB who gets a red test six months from now has no idea what's going on.

Worse, adding more to these mocks may cause some of the tests to be evergreen, and others to be required (deleting test 1 makes test 3 fail). Which is only discovered when a customer complains about bugs and you say, "wait, don't we have tests for this?" The tests are coupled which is bad juju. Every test should be capable of being run in isolation (debugging your tests!)

Often the first thing I do is rip this mess out and winnow the mocks down to the one, two, or three calls that actually need, so each test can be taken at face value. I also replace a lot of ten line tests with more four line, transitive tests (if A->B and B->C, then A->C.) Then I remove any coupling between the input and output that wasn't fixed by doing that (once a month almost, I find a test whose assertion boils down to expect(a).toEqual(a)). And the utility functions get split up into matchers with much more diagnostic data added in, and a bunch of incidentals taken out.

Effectively what we are doing versus languages where writing the code up front requires more thought, is that we are marking stories as done that aren't properly done for weeks, months, or years. It's a bit of knowledge that makes me uncomfortable. I have a lot easier time hiding that discomfort with people whose position in the org chart implies that they are still learning, and quite a time being constructive to the braggadocios. It's a lot of emotional labor and it's taxing.