[godelski]

> Even worse, occasionally I will accept a suggestion and only later notice a subtle mistake.

This is what hits me hard, as when I write it myself these things are more clear.

> I see people on Twitter and Hacker News talking about how much their productivity is being boosted by Copilot and ChatGPT…

I use ChatGPT quite a lot, but not really for coding. It does increase my productivity, but only because it better acts as a fuzzy search/query machine than Google. Often I use the two together (which leads to using Bard). I can remember the concept, but not the name, so GPT/Bard tells me the name, and then I can google from there. With this type of searching the hallucinations are not a big issue as they are only minor interruptions and are far outweighed just by the amount of searching I'd need to do to even find the thing I need in the first place. I should mention that I'm a researcher so I'm often looking for new tools, ways things have been done before, and frequently probing for things that are just outside my area of knowledge. But I should also say that I won't trust them to teach me a concept because when I've tested it on things I know, there are serious mistakes. Realistically the accuracy depends on how common/frequent the information is. If it is something that many people write about in a general field, it'll be accurate. If it is a topic, even popular, of a niche subfield, good luck.