[dkn]

I am curious in your experience how often the LLM must also update the tests. I find that if LLMs write tests after the implementation exists, they are either extremely brittle because they are coupled to the implementation, or they cover little of value because they mock everything to the point of testing nothing.

[mplanchard]

I have found a decent trick to be to write a parameterized test with e.g. a `cases` array that tests a function how you want it tested. Then ask the LLM to fill out more cases. It’s not perfect, but results in much less brittleness since you’ve already defined the specifics of what gets tested and what doesn’t.

[dcastm]

Same for me. I actively ask the LLM to write as few tests as possible. Otherwise you end up redundant and low value ttests.

[dkn]

Yep, and wasted token spend on an ongoing basis.

I instruct the LLM to follow TDD practices in certain areas, but otherwise prioritize integration style tests at the edges.