An LLM is a big equation that we solve to get textual output. If Ai proponents already believe an equation can contain consciousness, what about the Chinese Room presents a more compelling counterargument?
I'm not bringing up the Chinese Room in its original sense of asserting that a machine can't have consciousness. Indeed, I have no reason to suspect that human consciousness is irreproducible. Rather, the point here is to emphasize that our methods of determination are insufficient. A man armed with an English-to-Chinese dictionary appears to know Chinese until he's faced with an instance of linguistic ambiguity outside the scope of what a dictionary encodes; in other words, you can, with enough probing, disprove that the room understands Chinese. But how do you prove it in an affirmative sense? Let's use a simpler example: does Hans the Clever Horse understand arithmetic? We can disprove it by throwing enough arithmetic at Hans to prove that his understanding does not generalize, but if Hans did know arithmetic, how would we prove it? There are plenty of things in this category--things that we believe to some confidence level, but cannot prove--but let's frame it for what it is: belief and faith, not proof or logic. Whatever utility I may derive from LLMs, I have every reason to be skeptical of a movement of people whose awe of LLMs echoes the steadfast furor of the religious adherent.