[gorjusborg]

This exactly.

I know human nature, and that is what is concerning about AI.

Right now, most people have lived in a world where LLMs didn't exist. They know not to trust them as an oracle of truth. They know they are flawed and have counter-examples to back up any claim that responses are incorrect. They will second-guess the results, and may consider that the results could biased based on a flawed or limited data set.

If LLMs stick around, this situation may be different. As technology matures, it gets incorporated into living and thoughts of its users. Distrust transitions into trust given exposure and time. People forget the basis (and of the limitations inherent to) the underpinnings of the tech they use all the time.

That's what terrifies me: a world where people stop actively thinking and delegate decision-making to a language model built from flawed data. It's just so convenient to delegate. Not questioning is so much easier than questioning.