[korhojoa]

LLM's can be good at guessing the code you want, but I've seen them guess the comment or documentation correctly, then the implementation is the opposite of what you asked. It also generated tests which were for the incorrect case, with comments showing the opposite of what you wanted.

For somebody new at the language or new to programming in general, that difference can be imperceptible until it gets to a review.

I've also managed to spot people using an LLM to write stuff due to them doing things which won't pass linters or code tests.

[joshstrange]

The subtle differences between comment/code can absolutely be a problem, I’ve fallen prey to it as well but as long as you test/execute the code you’ll catch it if you didn’t catch it from reading over the code. I often reject anything overly clever or that I can’t reason out quickly when it comes to LLM generated code.

As for not passing linters or code tests that’s unacceptable behavior from an employee. If that happened more than a few times I’d be having a talk with the employee followed by showing them the door if they continued to do it. I’d behave the exact same way if no LLM was involved.

I do wish LLMs like Copilot could use the lint rules and/or tests to inform the code they generate but thankfully it’s normally a keystroke away for me to reform a block of code to match the style I prefer so it not the end of the world.

To use LLMs (or any tool) you have to be able to show you can use it correctly, safely, effectively, etc. If you can’t do that then you have no business being in a coding role IMHO.