[netsectoday]

Here are some indicators I've seen from 10x and 100x:

The 10x person fits very well in their organization. They are admired by upper-management for their ability to quickly comprehend, estimate, and deliver solutions to problems. They mentor everyone that asks or needs help. They provide 95% test coverage on all code. They show up for work well dressed, clean-cut, and on-time every day. Their code isn't 'brilliant' because they understand it must be maintained by the whole team. Their code reads well across difference experience levels but also includes clever tricks. They have an absolutely rock-solid understanding of the operating environment, platform, and modules. They don't document their code. They are the smartest person you know.

The 100x person seems to fit very well into the organization at first, and appears to be a 10x. They take on several seemingly-impossible tasks up front and eat it for breakfast. They automate large portions of the business and their own processes. They provide 99% test coverage on all code. They create production-quality docs after their public interface solidifies. They know security and implement it correctly. They spend months to years on self-driven projects to master different categories and build their tech-tree. They have invented tech and patented it. They bump into the intentional barriers of the organization. They break through and create a billion-dollar product, or they are removed from the organization. They are the smartest person you know and without explanation they appear to be insane.

It really seems 10x is a stigmatized term. It used to mean a developer that could produce 10x lines of code, but at this point I've taken it to mean a single developer that can leverage technology, their department, and organization to manifest a 10x positive impact compared to someone just cashing their check delivering story points. I don't think this term really fits for someone on the spectrum wreaking havoc in their organization.

[jmchuster]

in my experience, 80% coverage is about the correct number, but then those same lines of code are being covered over and over again for each major business use case.

[netsectoday]

Bumping the test coverage up past 90% definitely has lines re-run multiple times, with core logic even being re-executed possibly hundreds of times. This is absolutely ok! Your main purpose of writing tests is to make sure when you change business logic that a test will point you to the exact spot. If you commented out any single line of business logic; a test should fail.

I understand that this may seem crazy if you are used to 80% coverage, however there are subtle ways to build your unit/integration/feature/security/performance tests where you only pin your public api, and code changes to the internals don't break the tests, and the test suites only have minimal (necessary) overlap. This is test engineering and it takes about a decade of TDD to be comfortable with.

I would have said 100% test coverage for 100x but that last 1% isn't worth it. Some things just can't be tested without altering the public interface of your code JUST to accommodate the test, or they require some crazy significant re-work of the code that makes it hard to follow, or you have a module with a crazy way of interfacing with it that makes testing a specific part extremely difficult, or it's code that doesn't really need a test (like a log statement).

Side note: 99% test coverage isn't typical but is absolutely necessary if you have automatic patch-management solutions in place and are striving towards zero-defect code. How comfortable would you be pushing to production after bumping the major version of your platform and seeing your test suite pass? Having 99% confidence in this situation is a good thing and saves you time in the long run.

[hinkley]

I begin to relax around 85%.

It’s perfectly possible for a project to hit 85% without much untoward going on. Running the code and testing the code are not the same, and as the numbers go up I tend to find shenanigans.

I’d rather have an honest 85% than any dishonest number you could want. My butt starts to pucker again above 95%, because it’s all lies.

There also comes a point where running certain tests by hand is faster or more reliable than having a test for it. False positives (failing tests) are expensive, and in a way that is often counted poorly.

For a start, a false positive doesn’t cost the time of the test, it costs the time of the entire test suite times three - the one you wasted, the one you do over, and time between getting a failure and noticing the failure.

It also costs trust in the tools. Red tests become untrusted, which really hurts when you have a test that has false negatives. People become conditioned to hit the run button again without understanding the error, which then blows up farther down the line, possibly in production.

[netsectoday]

> My butt starts to pucker again above 95%, because it’s all lies.

With the way you are describing false-positives; it sounds like most of your tests are written as integration or feature specs. Exercising the UI at 95% coverage is insane (maybe that is why you said it's all lies). You will definitely be pulling your hair out with false-positives and with a large enough project you'll never get it 100% passing each day.

The answer here is to shift your tests from the UI or integration level down to unit tests. You should test at the function or model level with unit tests to pin all possible execution paths, then perform a sanity check test in the UI. For example; exhaustively test your login module at the unit level, then perform a sanity check at the UI level. Moving exhaustive tests down to the function or unit level increases test speed because you don't have to load or render the UI to perform the tests (which also eliminates most classes of false-positive tests).

If you or your team have issues with false-positives at the unit level then something isn't right with your test harness, OR you need to git-blame to identify the person on your team that needs some mentoring.

[hinkley]

Testing is hard. Most of the people who tell you otherwise are full of... bravado. Sometimes they have the worst tests.

When I started working with more dynamic languages I took comfort in the promises from TDD folks that you can write tests to cover all of these things, but I've seen how tests are written in practice and it's not good. I've met maybe ten people in twenty years who are really competent at writing bulletproof tests (as in, "You wrote tests for that? Enough said" versus Trust but Verify) and I am not at the top of that list. Frequently enough, I find eyerollers that have my name by them in the commit history.

I'm not saying I'm ready to go back to static languages because of this, but I will say that I see evidence for hope on the static analysis side for once in a very long time.

As you say, the problem is often discerning between unit tests and functional tests. Some people clearly can't tell the difference, but a number of people have confessed discomfort at writing unit tests (something about my frank communication style seems to invite people to share their fears with me, and I hear things that the "everything is great" people don't due to self-censorship). That these unit tests seem juvenile, pedestrian. Remedial. But to my mind that's the point. The cognitive load of determining why I just caused a bunch of tests to go red should be near zero, since I'm still trying to keep my head wrapped around the new code I'm in the midst of writing.

For about as long as I've been writing tests I've worked with people who think writing their own mocks or helper functions in tests is a good use of time. They take some work, so once you have them working they write five more tests that use them. The poor SOB who gets a red test six months from now has no idea what's going on.

Worse, adding more to these mocks may cause some of the tests to be evergreen, and others to be required (deleting test 1 makes test 3 fail). Which is only discovered when a customer complains about bugs and you say, "wait, don't we have tests for this?" The tests are coupled which is bad juju. Every test should be capable of being run in isolation (debugging your tests!)

Often the first thing I do is rip this mess out and winnow the mocks down to the one, two, or three calls that actually need, so each test can be taken at face value. I also replace a lot of ten line tests with more four line, transitive tests (if A->B and B->C, then A->C.) Then I remove any coupling between the input and output that wasn't fixed by doing that (once a month almost, I find a test whose assertion boils down to expect(a).toEqual(a)). And the utility functions get split up into matchers with much more diagnostic data added in, and a bunch of incidentals taken out.

Effectively what we are doing versus languages where writing the code up front requires more thought, is that we are marking stories as done that aren't properly done for weeks, months, or years. It's a bit of knowledge that makes me uncomfortable. I have a lot easier time hiding that discomfort with people whose position in the org chart implies that they are still learning, and quite a time being constructive to the braggadocios. It's a lot of emotional labor and it's taxing.