[tablarasa]

> How does the value of LLM-produced 100% branch coverage compare to no-LLM 100% branch coverage?

This is such a salient question. Sometimes (definitely not always) the test suites produced by LLMs are so trivial it's scary. Coverage can be an illusion for sure.

[coldtea]

I have been seeing such trivial shit getting submitted and deleting them from the test suites, but it's disheartening to see that the devs didn't have the better taste to see those LLM "tests" are so trivial they border on noop before submitting them...

[westurner]

What new or existing metric would better assess test coverage than line coverage and branch coverage?

I wrote a tool called tert - I guess it's called an agent harness now - to run various test runners and log test output and coverage output to disk. FWIU stripping spaces from JSON does save tokens. It seems like feeding coverage lines-missing maps into the prompt results in better output, better LLM-authored tests.

"Refactor these tests for maintainability and coverage. Use fixtures, mocks, and parametrization"

[coldtea]

>What new or existing metric would better assess test coverage than line coverage and branch coverage?

Substance coverage - testing the actual logic, edge cases, etc. Not mere lines.

[what]

You can write tests that cover every line/branch without actually verifying the code does what it’s supposed to. Relying on (unreviewed?) LLM written tests that provide coverage may not be a great idea.

[westurner]

So, a project can have 100% test coverage (the tests run each line or branch at least once without any errors or failures) but fail acceptance tests.

If there is prompt insufficiency, there is probably acceptance test insufficiency.

A more assuming agent could automatically develop a plan that includes presumptive acceptance tests and request feedback before spending tokens

[mkw5053]

We try to use types as much as possible. And if we can't, then lint rules, etc. Basically, get everything as far left in the SDLC as we can.

And, if/where we need tests, we write the source so they are few, high value, and complementary. Like actual unit tests, not complex with stuff like mocks just to generate trivial coverage.