[scubbo]

> If its working code and indistinguishable from a human answer to anyone reading it

These are extremely stringent conditions, though. Convincing-looking-but-wrong answers would arguably be even more damaging than a lack of answers.

(I suppose you could argue that these could arise from humans as well as from AGIs. I'm not sure if there's a good counter-argument to that)

[sgc]

The bot would need to learn as well as a "reasonable human" from being corrected on SO, and also be able to react in a socially appropriate way to correction (both in the subject thread and in future postings), otherwise it is a downgrade, even if initial answer is identical.

[hxugufjfjf]

My experience with OpenAI is that it is very good at exactly this, because it is so good at understanding context and follow-up questions. I was able to make it produce code that appeared correct, but was basically pseudocode with correct syntax, so it compiles/runs, but does essentially nothing. However, when prompted to actually make working code and explain how and why it works, it does so. And its also socially appropriate, not rude and what else you could/would expect when being called-out or corrected on its bullshit. I can only imagine future versions of the current AI model will be even better at this.

[hxugufjfjf]

I think your two final sentences capture the essence of what I was going to respond. "It doesn't matter if the answer is convincing-looking and wrong. It needs to work / be syntactically correct at a minimum, which OpenAI seems good at. However, the OP and others needs to test and evaluate if the proposed answer solve the original problem. And if its not, it will quickly be revealed as such, and "downvoted" or whatever stackoverflow functionality exists to indicate bad answers. This applies to both human and AI-generated answers."

[scubbo]

Yeah, absolutely. My position on AGIs for a long long time has been that they're great tools for a) generating insights into a large amount of data very quickly, and b) generating new instances of <thing> from examples of <thing> to help with exploring the possibility space of <thing>; but that any output or conclusion they generate _must_ be checked by a Human In The Loop, or at the very least their actions must be reversible without damage in the case of error.

[troupe]

Good point. It isn't like it is hard to find wrong answers on SO that were written by (hopefully) well meaning humans.