I think you're giving too much credit to the average user, empirically, we as humans tend to question less and believe more about things that are beyond our understanding, and LLMs are certainly way more of a black box than a search engine. The idea of a search engine is intuitive enough, but a large language model which involves maths that people likely haven't even heard of? and is the next big thing(TM) and will revolutionise the world? Yeah I have a feeling it will take some time before the average user understands the ins and outs of this tech.
Also the problem isn't necessarily about understanding the tech, but more about how it will be perceived at first. As long as every big player hypes up the tech as if it's perfect and revolutionary, I can't blame people for blindly trusting it at first(because it might as well work for most things they use it for, it will take some time before the pitfalls become apparent).
This. I don't think people lower their thresholds for applying critical thinking / fact checking just because it is an LLM's answer.
If someone blindly trusts and uses an LLM without checking for more evidence, then they would've done the same with any other source, be it Google, the newspaper or whatever.
You know, like lawyers that present motions full of made-up case law in court?
"In a cringe-inducing court hearing, a lawyer who relied on A.I. to craft a motion full of made-up case law said he “did not comprehend” that the chat bot could lead him astray." [1]
Also the problem isn't necessarily about understanding the tech, but more about how it will be perceived at first. As long as every big player hypes up the tech as if it's perfect and revolutionary, I can't blame people for blindly trusting it at first(because it might as well work for most things they use it for, it will take some time before the pitfalls become apparent).