[Fraterkes]

The assumption of this sort of argument, if I understand it correctly, is that consciousness is just an ordinary byproduct that appears (or grows gradually) somewhere on the spectrum of complexity.

If that’s true, mastering language is basically orthogonal to being conscious (as you’d maybe expect, GPT-2 was pretty good at language, but had a relatively tiny amount of “neurons”).

The same is arguably true for world modeling: a math textbook has a very complex and coherent model of a world, and is, probably, unconscious.

Then the question is: what reason do we have to assume that these models have some form of consciousness?

[lowbloodsugar]

Likewise, what reason do we have to assume that any given human is conscious?

[Fraterkes]

I don’t know why I’m consciousness, but I know that I am. Other people are outwardly very similar to me, so I assume the same is broadly true inwardly.

A rock is outwardly pretty similar to me; in that it exists in the world, has obvious physical boundaries, and is affected by the passage of time. But it is not so outwardly similar that I assume consciousness.

An LLM is also outwardly similar to me, in that it can express itself in language, and seems to ‘contain’ notions about the world. But it is again not necessarily so similar to me, outwardly, that inward similarity is obvious (to me)

[lowbloodsugar]

People talk to you, and you assume the are conscious. Computers talk to you, and you assume they are not conscious. The difference is that they aren't similar to you in certain metrics. This argument has been used before, in history, about people.

I find LLM frighteningly like many humans I meet: spouting "facts" that they have been taught. Like humans, when I ask an LLM to engage in critical thinking, the more advanced ones can.

[__MatrixMan__]

Or that anything at all is not conscious?

[Fraterkes]

Because we are, in our life, not always conscious.

[__MatrixMan__]

When you say "we" you're referring to something besides our minds?

Because sure, when we get knocked out, we're out. That's not an observation, it's a tautology. But there's nothing indicating that whatever remains doesn't have its own kind of consciousness.