[stratos123]

Such an argument is valid for a base model, but it falls apart for anything that underwent RL training. Evolution resulted in humans that have emotions, so it's possible for something similar to arise in models during RL, e.g. as a way to manage effort when solving complex problems. It's not all that likely (even the biggest training runs probably correspond to much less optimization pressure than millenia of natural selection), but it can't be ruled out¹, and hence it's unwise to be so certain that LLMs don't have experiences.

¹ With current methods, I mean. I don't think it's unknowable whether a model has experiences, just that we don't have anywhere near enough skill in interpretability to answer that.

[nothinkjustai]

It’s a completely different substrate. LLMs don’t have agency, they don’t have a conscious, they don’t have experiences, they don’t learn over time. I’m not saying that the debate is closed, but I also think there is great danger in thinking because a machine produces human-like output, that it should be given human-like ethical considerations. Maybe in the future AI will be considered along those grounds, but…well, it’s a difficult question. Extremely.

[Kim_Bruning]

What's the empirical basis for each of your statements here? Can you enumerate? Can you provide an operational definition for each?

[nothinkjustai]

Common sense.

[mrob]

It's plausible that LLMs experience things during training, but during inference an LLM is equivalent to a lookup table. An LLM is a pure function mapping a list of tokens to a set of token probabilities. It needs to be connected to a sampler to make it "chat", and each token of that chat is calculated separately (barring caching, which is an implementation detail that only affects performance). There is no internal state.

[stratos123]

The context is state. This is especially noticable for thinking models, which can emit tens of thousands of CoT tokens solving a problem. I'm guessing you're arguing that since LLMs "experience time discretely" (from every pass exactly one token is sampled, which gets appended to the current context), they can't have experiences. I don't think this argument holds - for example, it would mean a simulated human brain may or may not have experiences depending on technical details of how you simulate them, even though those ways produce exactly the same simulation.

[mrob]

The context is the simulated world, not the internal state. It can be freely edited without the LLM experiencing anything. The LLM itself never changes except during training (where I concede it could possibly be conscious, although I personally think that's unlikely).

[Kim_Bruning]

Right, no hidden internal state. Exactly. There's 0. And the weights are sitting there statically, which is absolutely true.

But my current favorite frontier model has this 1 million token mutable state just sitting there. Holding natural language. Which as we know can encode emotions. (Which I imagine you might demonstrate on reading my words, and then wisely temper in your reply)