[fluoridation]

>Maybe. How do you tell? What would you expect to be different if they didn't?

I think you're asking how I would know if other people were P-zombies. That's an inappropriate question because I didn't talk about subjective experience, just about internal state. There's no question about whether other people have internal states. I can show someone a piece of information in such a way that only they see it and then ask them to prove that they know it such that I can be certain to an arbitrarily high degree that their report is correct.

Unvoiced thoughts are trickier to prove, but quite often they leave their mark in the person's voiced thoughts.

>Insight is not solely a function of available input information. Arguably being able to search and extract the relevant parts is a far more important part of having insights.

LLMs are notoriously bad at judging relevance. I've noticed quite often if you ask a somewhat vague question they try to cold-read you by throwing various guesses to see which one you latch onto. They're very bad at interpreting novel metaphors, for example.

[lmm]

> I didn't talk about subjective experience, just about internal state. There's no question about whether other people have internal states. I can show someone a piece of information in such a way that only they see it and then ask them to prove that they know it such that I can be certain to an arbitrarily high degree that their report is correct.

Well, sure, but that much is equally true for an LLM with a scratchpad or what have you. (I guess you could say that the user should have access to the LLM's scratchpad and therefore be just as able to understand the state as the LLM itself, but as we move towards the LLM using its own state vectors that's less and less true in practice). I agree that a human may have a mood or secret knowledge or what have you in a way that an LLM wouldn't, but if all you're positing is access to some inert but hidden state then that feels like a Toaster-Enhanced Turing Machine.

[fluoridation]

>if all you're positing is access to some inert but hidden state then that feels like a Toaster-Enhanced Turing Machine

I thought it was pretty clear, given the context. What I'm saying is that humans are capable of limited introspection in ways that LLMs are not. They can remember their thought processes and review them ex post facto to answer questions that LLMs cannot. An LLM fundamentally cannot truthfully answer questions such as "why did you do this?" because its entire working memory is held in the context window. It doesn't know to any greater degree than you because it has no more information than you do; just like they are for you, its internal workings are a mystery. I'm not saying LLMs conceptually could not be designed with capabilities similar to a human's in this regard, with some symbolic memory that's capable of some bookkeeping, I'm saying none of the current ones have them.

[lmm]

> I didn't talk about subjective experience, just about internal state. There's no question about whether other people have internal states. I can show someone a piece of information in such a way that only they see it and then ask them to prove that they know it such that I can be certain to an arbitrarily high degree that their report is correct.

> What I'm saying is that humans are capable of limited introspection in ways that LLMs are not. They can remember their thought processes and review them ex post facto to answer questions that LLMs cannot.

But now you're making a much stronger claim than merely saying that internal state exists. Humans are capable of telling you a story about what their thought process was (as are LLMs). But whether that story will be accurate, much less contain new insights, is much harder prove.

[fluoridation]

>But now you're making a much stronger claim than merely saying that internal state exists.

It's not a different claim, it's the same claim. The reason humans are able to introspect is because they have that internal state.

>Humans are capable of telling you a story about what their thought process was (as are LLMs)

No. Humans can tell a story that's informed by introspection, while LLMs can only tell a story without any introspection. Humans may also lie and fabricate, but they are at least capable of introspecting, while LLMs are not.

>But whether that story will be accurate, much less contain new insights, is much harder prove.

If you're going to doubt the explanation then what's the point of asking the question? Necessarily it's going to be information that exists only in that person's mind, so at best you can check it for consistency with the person's own behavior and with the report itself, but some things you'll just have to either accept or ignore. Like, fundamentally you're asking the person to describe features of their own mind such as "he gets bored easily", "he can only hold so many facts at once", "he makes worse decisions under pressure", etc. If for example you're asking the question to improve something in the future (such as documentation or some procedure), it doesn't even make sense to distrust such reports, unless you believe a person like the one being described by the explanation doesn't and can't exist.

[lmm]

> It's not a different claim, it's the same claim. The reason humans are able to introspect is because they have that internal state.

> No. Humans can tell a story that's informed by introspection, while LLMs can only tell a story without any introspection. Humans may also lie and fabricate, but they are at least capable of introspecting, while LLMs are not.

There's still a gap here between "has some hidden internal state" and "that state can provide insight into to their thought process". If all you've shown is that knowledge that is public in LLMs is hidden in humans, there's no reason that should make the human better at introspecting (rather, it just makes the human harder to understand from outside).

> what's the point of asking the question?... If for example you're asking the question to improve something in the future (such as documentation or some procedure)

Indeed. If we knew that asking this kind of question of a human was more likely to provide insights that improved the process in the future than asking it of an LLM, that would be interesting. But it's quite a leap from "humans can have internal state" to that.

> unless you believe a person like the one being described by the explanation doesn't and can't exist

Meaning that a plausible explanation is valuable regardless of whether it's true? Wouldn't that apply just as well to an LLM's explanation?

[fluoridation]

>There's still a gap here between "has some hidden internal state" and "that state can provide insight into to their thought process".

No, because that internal state is part of the thought process. That's the whole point. You ask the human a question to learn something that you don't already know. It makes no sense to ask an LLM that because it knows nothing you don't already know; you and the LLM are privy to the exact same information. What's tripping you up about this?

>If we knew that asking this kind of question of a human was more likely to provide insights that improved the process in the future than asking it of an LLM, that would be interesting.

So, at this point I must ask: are you an NPC? Do you go through life just reacting to stimuli like a cockroach, with no understanding of why or how you do anything? If you're playing chess and someone asks you about a move you just made you are unable to explain, "I noticed such-and-such so I decided the best course of action was so-and-so to prevent this-and-that"? This is an alien concept to you? If so, then I'm sorry; most of us do not experience our own cognition in this way. We can perceive the formation of our own thoughts as well as the progressive retrieval of information.

>Meaning that a plausible explanation is valuable regardless of whether it's true? Wouldn't that apply just as well to an LLM's explanation?

See first paragraph.