[thisoneisreal]

It's a really fun philosophical exercise to ask what it means for them to be "wrong." My perspective is that they are fantastic at association and generalization (of language and symbols in particular), but whether they're identifying the associations you care about or generalizing to the level of abstraction you're aiming for is a complete crapshoot. If you aren't checking and correcting them, and discarding the misfires, you will end up with a very pretty Tower of Babel.

[al_borland]

One area where I feel safe saying they are “wrong”, rather than just going with a different assumption that was left unsaid, would be when it makes up API endpoints. It sees the general pattern in an API, then makes up an endpoint that sounds good, follows the pattern, but isn’t actually implemented.

I’ve also seen a lot of issues with co-workers using an LLM to write their readme files. I look at the readme for what return values I should get, go to use them, and get an error. I check the code, and sure enough, none of the variables in the readme exist. The LLM just through they sounded good. Things like this I would say are pretty objectively wrong.