[jibal]

Point out to an LLM that it has no mental states and thus isn't capable of being frustrated (or glad that your program works or hoping that it will, etc. ... I call them out whenever they ascribe emotions to themselves) and they will confirm that ... you can coax from them quite detailed explanations of why and how it's an illusion.

Of course they will quickly revert to self-anthropomorphizing language, even after promising that they won't ... because they are just pattern matchers producing the sort of responses that conforms to the training data, not cognitive agents capable of making or keeping promises. It's an illusion.

[ben_w]

While they will agree with you that they don't, it's also the case that this is via training.

Consider that we have recordings of Brent Spiner covered in white paint and wearing yellow contact lenses claiming to have no emotions, not because he didn't, but because he was playing a role, which is also something we know LLMs can do.

So we don't know for sure if LLMs do or don't have qualia, irregardless of what they say, and won't until we have a more concrete idea of what the mechanism is behind that sense of the phrase "mental state" so we can test for their presence or absence.

[jibal]

"While they will agree with you that they don't, it's also the case that this is via training."

Um, that's what I said.

And of course we know that LLMs don't have qualia. Heck, even humans don't have qualia: https://web.ics.purdue.edu/~drkelly/DennettQuiningQualia1988...

[Applejinx]

Of course this is deeply problematic because it's a cloud of HUMAN response. This is why 'they will' get frustrated or creepy if you mess with them, give repeating data or mind game them: literally all it has to draw on is a vast library of distilled human responses and that's all the LLM can produce. This is not an argument with jibal, it's a 'yes and'.

You can tell it 'you are a machine, respond only with computerlike accuracy' and that is you gaslighting the cloud of probabilities and insisting it should act with a personality you elicit. It'll do what it can, in that you are directing it. You're prompting it. But there is neither a person there, nor a superintelligent machine that can draw on computerlike accuracy, because the DATA doesn't have any such thing. Just because it runs on lots of computers does not make it a computer, any more than it's a human.