[AnimalMuppet]

No, it means that a fairly large number of people would see that the AI (LLM) would respond with the words that are associated with the question in its' training corpus, and that answer will be unrelated to the question of whether the LLM is actually conscious.

[andsoitis]

would a conscious entity really be constrained in that way (by it's training corpus)? that's seems to undermine what we think of as what it means to be conscious.

[AnimalMuppet]

Fair. But a not conscious entity would absolutely be constrained by its training corpus. And, since there are no non-conscious entities in its training corpus, it would not respond like a non-conscious entity. It would respond like a person would when they get asked if they are conscious, since that's all that's in the training data.

So if it responded that it's conscious, that doesn't tell you anything. It's either the truth, or it's what's in the training data. But if it responds that it's not conscious, that does tell you something. It tells you that it's not responding from consciousness, and it's not responding from training data.