[ben_w]

I've thought of the same analogy, but you know, I've never actually seen someone go "It's wrong about the stuff I understand, but I'll trust it anyway on everything I know nothing about". It's either:

(1) people getting caught using it to do their own jobs for them (i.e. they don't even realise it's wrong about the stuff they do understand);

(2) people who see the problems and therefore don't trust them anywhere at all (i.e. no amnesia, quite sensible reaction);

(3) people who see the problems and therefore limit their use to domains where the answers can be verified (I do this).

--

As an aside, I'm a little worried that I keep spotting turns of phrase that I associate with LLMs, for example where you write "you're absolutely right": I have no idea if that's all just us monkeys copying what we see around us (something we absolutely do), or if you're using that phrase deliberately because of the associations.

The only thing I'm confident of is that you're not doing is karma-farming with an LLM, but that's based on your other comments not sounding at all like LLMs so why would you (oh how surprising it was when I was first accused of being an LLM), but eh, dead internet theory feels more and more real…