[taberiand]

I believe that's called mutation testing. Using an LLM to perform the mutation sounds like a great idea

[rgmerk]

LLMs are not suitable for mutation testing. Mutation testing needs to be fast to be useful (because you need to generate and test a lot of mutated versions); an LLM-based mutator would be extremely slow as well as error-prone.

[taberiand]

Set aside LLMs, why does mutation testing need to be fast? It would be fine to have mutation tests run slowly, out-of-band of the main CI pipeline. They aren't mission critical, they're smoke tests for your unit tests.

Also you only need to generate a set of mutations for any particular unit once, and then again when the test code or the code under test changes.

[rgmerk]

Because mutation testing (should) generate a lot of mutants, which you then need to run your unit tests against.

[taberiand]

Yes but my point is it doesn't have to do that fast, as in unit test speeds.