[iLoveOncall]

That's the "Chinese room" argument: https://en.wikipedia.org/wiki/Chinese_room

It has been very well discussed before I don't understand how anyone sane can argue against it (and indeed you can see on Wikipedia that the replies to it are sparse and don't make any sense).

[cbolton]

Well I don't understand how one can accept this argument. I mean if you believe in mind-body dualism it can make sense. But AFAIK Searle doesn't, instead he holds that there's something special about the brain biology that enables consciousness and that you won't find in a computer. I don't see why that would be the case if the computer can simulate the real world, and I find Searle's argument against simulation, that simulating rain doesn't make you wet, falls flat: it can make things wet in simulation, and if you connect it to sprinklers it will make you wet.

[iLoveOncall]

> simulating rain doesn't make you wet, falls flat: it can make things wet in simulation, and if you connect it to sprinklers it will make you wet

It's not simulating rain if it's making you wet by using sprinklers?

[cbolton]

I mean you can connect the computer to sprinklers that activate when the system detects rain in the simulation if that's what you're after (that was just an aside to note that of course you don't get wet from a simulation disconnected from the real world).

But I guess that was a distraction from the main point: If consciousness emerges from biological processes in the brain connected to the world with a body, why would it not emerge from a simulation of these processes connected to the real world with sensors and actuators?

It seems like circular reasoning to me: The simulation is not like the real thing because it lacks the special thing that enables consciousness (that's Searle's biological naturalism). And it lacks what enables consciousness because it's not the real thing (that's the weather analogy).

[sp1nningaway]

I like this bit from the wiki article:

Speaking through Zarubin, Dneprov writes that "the only way to prove that machines can think is to turn yourself into a machine and examine your thinking process", and he concludes, as Searle does, that "even the most perfect simulation of machine thinking is not the thinking process itself."

That feels similar to what Chiang is saying. The physical state of being a human is part of human consciousness. An LLM would require many more "levels" that haven't been achieved yet. An LLM with a body, sense organs, physical tools, social dynamics, etc. would all be steps on the path to a "conscious" LLM.

[Jtarii]

It's fairly obvious to me that the room as an entire system understands Chinese. The human in his thought experiment would just represent the small fraction of the brain that encodes/decodes symbols.

[twobitshifter]

Agreed, he’s saying if he’s working with a black box he doesn’t understand what it’s doing. Which is of course true. If the black box were a chinese person instead of a tool and he was writing down what they said, would he say the chinese person in the black box does not understand why they told him to write something? No, of course not.

Today we know the program to do this is always going to be inscrutable to Searle. His role is no different from someone using Claude to write an email.