[seanmcdirmid]

But conformance testing is where agents really excel at if you set things up right:

* Black-box testing: the agent writing the tests cannot see implementation and the agent writing implementation cannot see tests, they only agree on a spec and an interface (the minimum needed to write tests).

* Evaluate test coverage using code coverage, but when gaps are found communicate those gaps in terms of the specification.

Good specs should be grounded (complete and not ambiguous), they don't need to be formal. You should be able to re-run your agents when the spec changes on diffs to the spec, and if a change happens out of bad, you should have agents that go in and propose fixes to the spec. Since agents are doing deterministic codegen like a compiler would, this is all pretty straightforward.

You also need to consider public and internal specifications (the public specification being for reuse of the component), and you might test your integrating component with a test-double (built from the public specification alone) rather than the real component itself.

[sdevonoes]

> Good specs should be grounded (complete and not ambiguous), they don't need to be formal

IMHO, this is a mistake. I guess we play with it because there’s isn’t anything better nowadays. Writing and maintaining “specs” in plain english is painful.

[seanmcdirmid]

Natural language is incredibly expressive and fairly easy to read and write. Pick your favorite formal specification language...and you can just express some properties in them, and they are mostly niche at that. If you increase the expressiveness of your formal language too much, it just becomes code, then you are back to square one. LLMs are also incredibly proficient in processing natural language; i.e. writing, maintaining, and using a spec written in plain english is actually fairly viable with a modern LLM.

[crazygringo]

> Writing and maintaining “specs” in plain english is painful.

The subject of this entire post is development with agents. Writing specs in English is how you do that. If you don't like it, then this is probably not the right article for you to be commenting on.