[whittingtonaaon]

The question is whether there was ever a good reason to suggest that mind is computational, and I think philosophy (i.e. Searle) have shown that there wasn’t. A brain CAN compute (albeit very slowly), but that’s the end of the similarities.

The reason it’s not morally wrong, say, to recycle an old computer is that computers are not at all like human minds.

[eru]

Alas, if you refer to Searle's Chinese Room, that hasn't shown anything, if you just bite the bullet and accept that the Room is conscious.

[_zywo]

I like Scott Aaronson's response[1] to the argument:

> So, class, how might a strong AI proponent respond to this argument? Duh: you might not understand Chinese, but the rule book does! Or if you like, understanding Chinese is an emergent property of the system consisting of you and the rule book, in the same sense that understanding English is an emergent property of the neurons in your brain. Like many other thought experiments, the Chinese Room gets its mileage from a deceptive choice of imagery -- and more to the point, from ignoring computational complexity. We're invited to imagine someone pushing around slips of paper with zero understanding or insight -- much like the doofus freshmen who write (a+b)^2=a^2+b^2 on their math tests. But how many slips of paper are we talking about? How big would the rule book have to be, and how quickly would you have to consult it, to carry out an intelligent Chinese conversation in anything resembling real time? If each page of the rule book corresponded to one neuron of (say) Debbie's brain, then probably we'd be talking about a "rule book" at least the size of the Earth, its pages searchable by a swarm of robots traveling at close to the speed of light. When you put it that way, maybe it's not so hard to imagine that this enormous Chinese-speaking entity -- this dian nao -- that we've brought into being might have something we'd be prepared to call understanding or insight.

[1]: https://www.scottaaronson.com/democritus/lec4.html

[eru]

Yes. See also what I quoted from another book by Scott in https://news.ycombinator.com/item?id=31807695

[hgomersall]

There's an interesting thought experiment that goes the other way. What if we replace a single neuron in a conscious person's brain by a properly connected real-time neuron simulator? Would that affect the person's consciousness? What about two neurons? Etc etc. At what point, if ever, would a progressively more simulated brain no longer be the original consciousness?

[hoseja]

This may be a little deceptive depending on how you define "a properly connected real-time neuron simulator". Can it form new connections? Can it grow? Does it metabolize? Can it die? Etc...

[kertoip_1]

But does it really matter? If we can just replace, one-by-one all neurons in brain with artificial ones and that would not affect consciousness of the user, then we've just proved it's possible to have a thinking machine

[hoseja]

What if you replace all the neurons and the person is suddenly stuck with no short-term memory and aphasia? Clearly your replacement neurons are defective. At which point are they not? Is that point distinguishable from normal neurons?

[sudosysgen]

What is this counterfactual? We're talking about replacing the neurons one by one, slowly. When would the loss of memory happen?

[criddell]

Is a person with no short-term memory and aphasia still conscious?

[hgomersall]

These are all good questions and probably allow our realised thought experiment to gain more insight into what aspects are necessary for consciousness and what are just biological baggage.

[hoseja]

I suspect the line is very blurry indeed.

[simonh]

Exactly, replace the 'room' with a volume the size of a large planet. Have each symbol represent an atom in a brain. If you like, replace the person manipulating the symbols with a vast army manipulating the symbols according to the laws of physics and chemistry. The Chinese room argument simply reduces to an argument against materialism.

[meroes]

Searle is a physcalist though

[simonh]

Sort of. He thinks there's something special about biology, but doesn't seem to be able to explain what that might be and how this is different from dualism, so it's hard for me to take it seriously.

To my mind both dualism and biological naturalism are both positing some special woo needed for consciousness without saying what that woo is, or what it's like, or how it works, or anything about it at all.

[meroes]

It may be woo, but how I understand his approach is a rejection of dualism and the Cartesian theater. To commit to physicalism but lean very hard onto what can be asked of it. One must do some kind of move to escape the Cartesian theater. There are those who say consciousness is too hard, and those who say we can take a stab at it. In the latter camp I think he’s more on the right track than anyone major figure I know of.

[simonh]

The Cartesian theatre is the view that there is some special place in the brain that ‘contains’ consciousness. Searle thinks there’s some special biological machinery in brains that causes consciousness.

Thats not quite the same thing, but if there is any distinction it’s a pretty darn fine one, and it certainly doesn’t exclude a Cartesian theatre interpretation.

[aneil]

But the argument shows it's absurd to think the room is conscious. What if the person in the room takes a coffee break or goes on vacation? What connects the next computational step they do to the previous one and the next... which then somehow gives rise to a fragment of qualia. That seems like an impossibly complex and unlikely scientific theory.

[eru]

> What if the person in the room takes a coffee break or goes on vacation?

Regular meat-and-bone people lose consciousness all the time, and regain it later. No big deal.

> What connects the next computational step they do to the previous one and the next...

Whatever index card system or similar the operating procedure in the room prescribes for keeping track of state?

> which then somehow gives rise to a fragment of qualia. That seems like an impossibly complex and unlikely scientific theory.

We don't have any 'scientific theory' of qualia. We don't even know if they exist, or how they would manifest in the physical world.

Since we don't know much of anything, I don't know whether a fragment of a figment would be more or less weird than the figment itself. Or whether we would even have fragments.

It's probably too early to try to have a theory of qualia that would apply here?

[aneil]

> Whatever index card system or similar the operating procedure in the room prescribes for keeping track of state?

You're missing the point. A bit is just some electrons. It could be a scribble in a notebook. But consciousness integrates several pieces of information into a coherent experience. The bits in an index card system could as well be some scratchings of graphite in a notebook. How would consciousness arise from graphite in a notebook?

> We don't have any 'scientific theory' of qualia. We don't even know if they exist,

I differ on this. The only thing I know for certain the universe contains is qualia. You, the idea there is a "me", atoms, bits, axons and electric potentials are merely ideas, which "I" apprehend as qualia.

> or how they would manifest in the physical world.

Correct, that is the question. But the Chinese room thought experiment shows it's not merely by information processing. I mean, atoms in a room are processing information - they are computing the next state of all of the atoms in the room. Are they conscious? How about a subset of those atoms? Are those conscious in a different way?

The point is that the consciousness-is-computation idea is just too weak to even be a physical theory.

[aneil]

> Regular meat-and-bone people lose consciousness all the time, and regain it later. No big deal.

So what? There are lots of REAL physical processes that are disrupted in a human being when they lose consciousness.

The point of the Chinese Room is to show that information processing alone is insufficient for consciousness. For example is information processing happening when the person pauses for a minute - or not? For a second, for a millisecond? What about when the pen comes off the paper? What about when he's sharpening his pencil? How exactly does the consciousness=information processing idea work for these situations? It's a nonsense idea that doesn't hold up to careful inspection.

And it's exactly akin to saying we get nuclear power by simulating a nuclear power plant in a computer.

On the other hand, if we believe that consciousness is like an ordinary physical property of the universe, either emergent or fundamental, then it should be related to other physical properties, just as electromagnetism is to mass and energy.

[eru]

Information _is_ a physical property. That's what thermodynamics is all about.

[hoseja]

The conscious system is so high above the guy shuffling rulebooks and slips of paper it has no concept of him. A billion years might pass for the Room to experience a second. Just as we have basically no concept of the baroque quantum-molecular-cellular machinery of our brains. There are very roughly 10^15~18 light-sensitive molecules in your eye so you can see. With our best computers it's very hard to precisely simulate a single one of them. Just ponder the insane scale.

[GoblinSlayer]

It's just usual neurodegenerative disease. Happens all the time.

[meroes]

No, what happens is consciousness instantiated the whole setup: natural language, Chinese language, syntax, the room, computation, dictionaries, lookups, etc.

The whole thing is a giant computation. Yet computations are as mundane (lifeless) as an abacus or pen and paper. It is just wrong to thing the abacus is conscious. Same for the room.

[whittingtonaaon]

I know Searle replied to this, but don’t remember what he said, except it seemed a little ridiculous to talk about sentient rooms. I think it’s much more plausible to just claim that the people who wrote the instructions know Chinese.

[simonh]

>it seemed a little ridiculous to talk about sentient rooms

That's part of the sleight of hand employed in the argument, of reducing it to one person in a room with a table stacked with symbols in front of them. He's misdirecting our intuition with a trick of scale. If instead I said the "room" was the size of Jupiter, and it had a vast army of people manipulating the symbols, and each input and output took millions of years (or you speed up the rate of manipulation arbitrarily), all of a sudden it seems less implausible.

[bnralt]

I'd say the biggest slight of hand is that human's understand other humans "knowing" something as something happening internal to an individuals brain. Holding a biology book in front of you doesn't mean you know everything in the book. What you "know" are the things from the book that you've stored in your brain.

We can modify the thought experiment so that it's something internal to the persons brain. The person doesn't learn Chinese as most people do, but a computer rewires their mind to be the same as someone who did. Think of The Matrix, when Neo says "I know Kung Fu." Would people then say this person speaks Chinese? I imagine just about everyone would say yes.

[quickthrower2]

If true it means there is a fundamental limit of science to understand the nature of the universe.

Because if we can unify a theory of physics, we can build a simulation on a computer, which could simulate us.

The get out might be that the energy required to run a computer that can simulate us is more than the energy in the universe and so is practically impossible.

[Gare]

A human mind runs only on 12 watts of power. Even if simulation is billion times more inefficient, it's only 12 gigawatts of power, obtainable even today for state actors and large companies.

Not saying it's possible, just that on power requirements alone it should be feasible.