[gymbeaux]

The unit test example has been my team's experience as well. The unit tests look good on the surface, but their passing or failing has little predictive value on whether there are actually bugs in the code.

Some people have suggested you write the unit tests by hand to basically "check" the LLM's work and keep it honest, but to write good unit tests you have to understand the underlying code, which takes time (since you didn't write it), so to me this is another bullet point that suggests LLMs will eventually be relegated to "StackOverflow+" duty - give me snippets, but I'll still write effectively all the code.