[jacobr1]

LLMs as clearly doing more than just modeling words. In order to predict word placement, they need to build some kind of model, their latent space, of the types of things they are able to predict. Not really full world models yet - but they have decent "blog space" or "github project" models. And you can see this with multimodality, or non-text modality modals such as images and audio. They map from their latent spaces to the outputs. The fact that multi-modal systems can share the interior layers shows some kind of internal representation is created.

[adamzwasserman]

Or.. the LLMs are just "telescope" sufficiently powerful to detect the aligned internal representation that already exists.

A more powerful telescope does not "create" new galaxies.

[jvanderbot]

Love the analogy. I a lot of people had their mind blown by vector space representation of words. The idea that Female + King averaged out around "Queen". Words might foundational to concepts or just really-well-designed ways to transmit them.

[sometimelurker]

You have the word 'bot' in your name and judging by your comment history love using em-dashes. Are you a machine or not?

[jvanderbot]

I doubt I used emdashes, but I do us "-". I dont have emdash unicode bound to my keyboard yet. (edit: Hyphenation is not emdash, of course)

you can view my profile details freely and decide for yourself, and a quick look might even explain the "bot" suffix.

[sometimelurker]

Oh I figured there's a real human being named Josh Vander and youre his openclaw or something. why do you have the `bot` suffix?

[jvanderbot]

Because I spent 15 years working in robotics, and I'm a nerd who over indexed identity on that :)

[mhjkl]

This comment would go hard in a sci-fi novel 5 years ago

[adamzwasserman]

I can guarantee you that the comment from jvanderbot was not AI generated.

[Fraterkes]

The assumption of this sort of argument, if I understand it correctly, is that consciousness is just an ordinary byproduct that appears (or grows gradually) somewhere on the spectrum of complexity.

If that’s true, mastering language is basically orthogonal to being conscious (as you’d maybe expect, GPT-2 was pretty good at language, but had a relatively tiny amount of “neurons”).

The same is arguably true for world modeling: a math textbook has a very complex and coherent model of a world, and is, probably, unconscious.

Then the question is: what reason do we have to assume that these models have some form of consciousness?

[lowbloodsugar]

Likewise, what reason do we have to assume that any given human is conscious?

[Fraterkes]

I don’t know why I’m consciousness, but I know that I am. Other people are outwardly very similar to me, so I assume the same is broadly true inwardly.

A rock is outwardly pretty similar to me; in that it exists in the world, has obvious physical boundaries, and is affected by the passage of time. But it is not so outwardly similar that I assume consciousness.

An LLM is also outwardly similar to me, in that it can express itself in language, and seems to ‘contain’ notions about the world. But it is again not necessarily so similar to me, outwardly, that inward similarity is obvious (to me)

[lowbloodsugar]

People talk to you, and you assume the are conscious. Computers talk to you, and you assume they are not conscious. The difference is that they aren't similar to you in certain metrics. This argument has been used before, in history, about people.

I find LLM frighteningly like many humans I meet: spouting "facts" that they have been taught. Like humans, when I ask an LLM to engage in critical thinking, the more advanced ones can.

[__MatrixMan__]

Or that anything at all is not conscious?

[Fraterkes]

Because we are, in our life, not always conscious.

[__MatrixMan__]

When you say "we" you're referring to something besides our minds?

Because sure, when we get knocked out, we're out. That's not an observation, it's a tautology. But there's nothing indicating that whatever remains doesn't have its own kind of consciousness.

[sometimelurker]

And they do have internal latent vectors for their own states, so there's some kind of recursion/introspectiveness dynamic.

[dinfinity]

Exactly. The translation back to words is the final step, so in a way very similar to what the post describes.

Improvements in model performance have been made exactly by having intermediate steps stay in the form of internal representations rather than words.

[kibwen]

Except that the Chinese Room shows that the existence of a mapping from input to output, however emergently it might have been devised, is not alone sufficient to demonstrate understanding.

[SpicyLemonZest]

"Understanding" is your word, it doesn't appear in the source article nor the comment you're responding to. The Chinese Room argument does not attempt to show that a mapping from input to output can't implement a latent space, or that it can't implement complex models of what the language is describing. If a person can express those things explicitly in the output, or if you have to do them in order to correctly respond how a person would, then the room by definition has those capabilities.

(What's the point of the argument if it doesn't tell us anything about the capabilities or internals of an AI? I'm not sure.)

[jacobr1]

Right - the point I was refuting is that LLMs are "just statistical models of words." More is going on. Does that imply "understanding?" I don't know, I'm not sure we have a good enough definition to say. But it does mean that the models are more complex that say, markov chain graphs with corpus frequencies. It seems we are encoding data in the latent space with much higher complexity than "just words." There is higher order semantic information being captured - probably not the same has human "thoughts" - but again - also not _just words_.

[Marha01]

The Chinese Room as a whole understands.

[dinfinity]

The man in the room is comparable to a human hand, and the magic rulebook to a human brain.

This in the sense that we can easily retain human "understanding" by stripping away almost all parts of the human body or replacing them with fairly trivially made replacements, except for the brain.

In the Chinese Room the equivalent is the magic rulebook: We have no idea how to construct/replace it, yet people somehow handwave that away whilst simultaneously confidently asserting it does not understand anything.

[kibwen]

That's irrelevant to what we're talking about, though. The point here is not to assert whether or not the room understands, but to emphasize that our methods are insufficient to demonstrate this. You could posit that the room understands, or you could posit the reverse, and neither argument can be refuted.

[Fraterkes]

An LLM is a big equation that we solve to get textual output. If Ai proponents already believe an equation can contain consciousness, what about the Chinese Room presents a more compelling counterargument?

[kibwen]

I'm not bringing up the Chinese Room in its original sense of asserting that a machine can't have consciousness. Indeed, I have no reason to suspect that human consciousness is irreproducible. Rather, the point here is to emphasize that our methods of determination are insufficient. A man armed with an English-to-Chinese dictionary appears to know Chinese until he's faced with an instance of linguistic ambiguity outside the scope of what a dictionary encodes; in other words, you can, with enough probing, disprove that the room understands Chinese. But how do you prove it in an affirmative sense? Let's use a simpler example: does Hans the Clever Horse understand arithmetic? We can disprove it by throwing enough arithmetic at Hans to prove that his understanding does not generalize, but if Hans did know arithmetic, how would we prove it? There are plenty of things in this category--things that we believe to some confidence level, but cannot prove--but let's frame it for what it is: belief and faith, not proof or logic. Whatever utility I may derive from LLMs, I have every reason to be skeptical of a movement of people whose awe of LLMs echoes the steadfast furor of the religious adherent.

[PaulDavisThe1st]

It does so only in the claims of its creator. Plenty of other people have pointed out fallacies in the claim. My favorite is the Dennett/Hofstadter observation that while the man in the room may not understand chinese, the room/system certainly does.

[swid]

If our own brains were insufficient to demonstrate the fallacy of the Chinese Room; the particles, molecules, cells, synapses, and so forth of our bodies cannot be believed to have understanding, but we recognize it in the sum of the part anyway....

... we now have LLMs to even more pointedly show the deficiencies of the argument.