I fell for it a few minutes the other day. Debugging an issue with a device, the AI wrote "I have a strong hypothesis about the cause in the code". I asked it to write out the hypothesis & create a test plan to validate it. It made a test plan, but no hypothesis. The test plan did not reproduce the issue, and it turned out to be a hardware design problem not in the code at all. But for a moment in there I thought it actually had a hypothesis, I forgot that it's not thinking beyond what's written in the chat. Someone who was going to reproduce & fix a bug would probably write "I have a strong hypothesis about the cause" or similar, so it played along & wrote that.
If the hypothesis is not printed out in the context, then it cannot hold it past that turn. You could prompt it to generate said hypothesis first (or set of hypotheses), and only then act on them. And then things might work.
Definitely not exactly a human. OTOH Low hanging fruit is low.
You could reverse that argument. The only thing that ever happens in a human mind is a Sodium-Kalium semi-permeable membrane balancing out (meaning going from polarized to unpolarized) and triggering the tiniest of explosions spreading one of 4 chemicals around. Repeat a few billion times per second for ~80 years.
The Eliza effect is off the scale.
What I'm trying to say is that the underlying method is not a valid reason to discredit one thinking process over another.
I remain baffled that anyone thinks dragging brains into discussions of these things does anything but make everyone more confused. This kind of thing is exactly what I'm getting at—that it's common for even people in the computer technology field to think the comparison is apt, or illuminates anything, is a wild indication of how inclined we are to be tricked by computer programs that happen to operate on language.
You are baffled because of your own ignorance of the underlying principles under discussion. Do you believe in a dualist interpretation of reality, that the process of thinking is somehow nonphysical? That these programs operate on language is besides the point. The fact you think this is why it's interesting shows you don't even understand the argument.
Are you familiar with the physical church turing thesis?
The effect is not quite what you think it is, and people don't quite take the right lessons.
Similar to the eliza effect, people still take the original reading of Clever Hans: "he couldn't really do maths, he's just taking social cues from his handler"
But what's the actual difference between Eliza, Clever Hans and RLHF? They're doing the similar things, right?
Now look at how we valued that in the 20th vs 21st century:
How much does an ALU even cost anymore? even a really good one? (it's almost never separate anymore, usually on the same silicon as the rest of the cpu/microcontroller)
Meanwhile... what's the TCO to deploy a sentiment classifier? Especially a really good one?