[tel]

I disagree. "Given an input, respond as yourself" is what humans call talking. "Given an input, predict the most plausible continuation" is something else entirely.

To put a fine point on it, the AI has no state of mind, no allegiance to an identity. Asking only for "the most plausible continuation" is considerably more freedom, and more challenge, than humans perform in conversation.

[chpatrick]

> the AI has no state of mind, no allegiance to an identity

That fully depends on the context and the fine tuning that has been applied.

[tel]

We're getting kind of to the point where these words are hard to define, but it's worth really interrogating these. Ultimately, the context and the prompt don't change the fundamental nature of how the system was optimized.

You can try very hard to construct a prompt such that the most plausible continuations of that prompt are consistent with a notion of identity, but you can also easily witness pulling the AI out of that prompt. Jailbreaks do that work today.

But even then, it's, largely, continuing the prompt as well as possible. Many "identities" can satisfy that aim. See the idea of "Waluigis" for example.