[drmajormccheese]

There are techniques for improving our confidence in our software: unit testing, integration testing, fuzz testing, property-based testing, static analysis, model checking, theorem proving, formal methods, etc. The LLM is not only a tool for generating lines of code. It can also generate lines of testing. The goal is that the tests are easier to audit by the humans than the code.

[sfjailbird]

How do we make sure the LLM generated code works? We'll have LLM generated tests! Wait a minute...

[exographicskip]

I've found that one of the areas I enjoyed least is now what I spend a lot of time on now: testing!

Property-based testing in particular has uncovered a number of invariants in every code base I've introduced it to.

tbf depending on the agent/model a lot of the tests end up being thrown out so it's possible I _should_ handwrite more tests, but having better prompts and detailed plans seems to mitigate that somewhat

[coldtea]

>There are techniques for improving our confidence in our software: unit testing, integration testing, fuzz testing, property-based testing, static analysis, model checking, theorem proving, formal methods, etc. The LLM is not only a tool for generating lines of code. It can also generate lines of testing.

Which is the same issue of lack of understanding and care and accountability from the human operator, with extra steps and a false sense of security.