[westurner]

What new or existing metric would better assess test coverage than line coverage and branch coverage?

I wrote a tool called tert - I guess it's called an agent harness now - to run various test runners and log test output and coverage output to disk. FWIU stripping spaces from JSON does save tokens. It seems like feeding coverage lines-missing maps into the prompt results in better output, better LLM-authored tests.

"Refactor these tests for maintainability and coverage. Use fixtures, mocks, and parametrization"

[coldtea]

>What new or existing metric would better assess test coverage than line coverage and branch coverage?

Substance coverage - testing the actual logic, edge cases, etc. Not mere lines.

[what]

You can write tests that cover every line/branch without actually verifying the code does what it’s supposed to. Relying on (unreviewed?) LLM written tests that provide coverage may not be a great idea.

[westurner]

So, a project can have 100% test coverage (the tests run each line or branch at least once without any errors or failures) but fail acceptance tests.

If there is prompt insufficiency, there is probably acceptance test insufficiency.

A more assuming agent could automatically develop a plan that includes presumptive acceptance tests and request feedback before spending tokens