| I think humans are just hard-wired to interpret language and conversation as signs of intelligence. The way LLMs actually behave is counterintuitive in a way that violates most people's expectations. You can have a complex, rational conversation with it. Give it commands and have it appear to infer details and anticipate the future. Also, "AI" has been a part of pop culture and pop science for decades... people have been conditioned by Star Trek and Star Wars and the like to accept that a computer you can "just talk to" is self-aware, and that computers never make mistakes. It's a bias you even find in discussions about self-driving cars. Computers are perfectly logical, rational, mathematically precise and if they were to be self-aware, would hold to a perfectly mathematically correct and provable model of reality based only on concrete data, without any prejudice or bias. Commander Data can't lie, can't even use contractions. Of course we can trust them to drive our cars, we just can't trust humans. So we're left with the idea floating around in popular consciousness that whenever AI shows up, we should trust it implicitly. The one thing computers have never done (until now,) and certainly never been known to do, is make up completely plausible nonsense and gaslight you into accepting it. LLMs can't even do math. This isn't the way "computers" behave. One doesn't even really have to conflate any of this into religious belief for it to make sense. "LLMs being intelligent" is the model that just already happens to fit everyone's priors. It just happens to be dangerously wrong. |
...And just where do you think those computers came from, sparky?
Back to square one.
If you can't trust the humans in the chain, trusting the system as a whole is the act of a fool. You've abandoned rationality, and comfortably ensconced yourself in the plush comfort of irrationality and ignorance, setting yourself up to get rolled by the real master's in the house, who'll be the ones who make, run, and operate the machine. Double points for not even realizing the only one pulling the wool over your eyes is yourself.
You're trusting something that half the time needs effing cross-checking by a squishy bit due to bored programmers misusing computing primitives in ways in which their integrity and soundness is undermined by their physical implementation.