[vhold]

Part of the problem is ChatGPT’s incorrect answers are so well formed they seem more correct than the average incorrect human answer which is usually short, lazy, not even related to the question, contains no explanation, etc.

ChatGPT creates long logical answers often with explanations and the errors are very subtle. If it were let to run wild across stack overflow it would take a lot more effort to weed them out.

[piaste]

It's worse than that, ChatGPT's answer are superficial on a fundamental level.

Earlier today, I asked it to produce a complicated piece of code in a niche language, and it dutifully did so - but the code had a type error, and used a reserved keyword as a function name.

When I asked it to rename the function, it did so successfully, including usages and comments. (So it didn't totally treat the code piece as opaque.)

But when I told it, "there is a type error here in the function X, the type should be (this)" it replied: "You are right, I fixed the mistake. Here is the code but with the function X having type Y", followed by the original code verbatim with no changes whatsoever.

This really helped me understand what these text generation AIs do. They try to produce a text that looks like a correct answer, which may or may not be correct.

In my example, it clearly didn't have nearly enough training on the specific language and programming style to decompose and "tune" the code to the right answer. But it had a lot of training in saying "right, I fixed the bug, here's the correct code", whether or not it was true.