[apwell23]

i would love to used to use it change code in ways that compiles and see if test fails. Coverage metric sometimes doesn't really tell you if some piece of code is covered or not.

[sesm]

Coverage metric can tell if lines of code were executed, but they can't tell if execution result was checked.

[taberiand]

I believe that's called mutation testing. Using an LLM to perform the mutation sounds like a great idea

[rgmerk]

LLMs are not suitable for mutation testing. Mutation testing needs to be fast to be useful (because you need to generate and test a lot of mutated versions); an LLM-based mutator would be extremely slow as well as error-prone.

[taberiand]

Set aside LLMs, why does mutation testing need to be fast? It would be fine to have mutation tests run slowly, out-of-band of the main CI pipeline. They aren't mission critical, they're smoke tests for your unit tests.

Also you only need to generate a set of mutations for any particular unit once, and then again when the test code or the code under test changes.

[rgmerk]

Because mutation testing (should) generate a lot of mutants, which you then need to run your unit tests against.

[taberiand]

Yes but my point is it doesn't have to do that fast, as in unit test speeds.