[kalkin]

This basically Searle's Chinese Room argument. It's got a respectable history (... Searle's personal ethics aside) but it's not something that has produced any kind of consensus among philosophers. Note that it would apply to any AI instantiated as a Turing machine and to a simulation of human brain at an arbitrary level of detail as well.

There is a section on the Chinese Room argument in the book.

(I personally am skeptical that LLMs have any conscious experience. I just don't think it's a ridiculous question.)

[benzible]

That philosophers still debate it isn’t a counterargument. Philosophers still debate lots of things. Where’s the flaw in the actual reasoning? The computation is substrate-independent. Running it slower on paper doesn’t change what’s being computed. If there’s no experiencer when you do arithmetic by hand, parallelizing it on silicon doesn’t summon one.

[famouswaffles]

Exactly what part of your brain can you point to and say, "This is it. This understands Chinese" ? Your brain is every bit a Chinese Room as a Large Language Model. That's the flaw.

And unless you believe in a metaphysical reality to the body, then your point about substrate independence cuts for the brain as well.

[comp_throw7]

The same is true of humans, and so the argument fails to demonstrate anything interesting.

[akersten]

> The same is true of humans,

What is? That you can run us on paper? That seems demonstrably false

[kalkin]

If a human is ultimately made up of nothing more than particles obeying the laws of physics, it would be in principle possible to simulate one on paper. Completely impractical, but the same is true of simulating Claude by hand (presuming Anthropic doesn't have some kind of insane secret efficiency breakthrough which allows many orders of magnitude fewer flops to run Claude than other models, which they're cleverly disguising by buying billions of dollars of compute they don't need).

[benzible]

The physics argument assumes consciousness is computable. We don't know that. Maybe it requires specific substrates, continuous processes, quantum effects that aren't classically simulable. We genuinely don't know. With LLMs we have certainty it's computation because we built it. With brains we have an open question.

[kalkin]

It would be pretty arrogant, I think, though possibly classic tech-bro behavior, for Anthropic to say, "you know what, smart people who've spent their whole lives thinking and debating about this don't have any agreement on what's required for consciousness, but we're good at engineering so we can just say that some of those people are idiots and we can give their conclusions zero credence."