[lxgr]

> Asked about the answer, ChatGPT points to the instruction set and that it allowed it to add additional statements: [...]

I don't think this is how this works. It's debatable whether current LLMs have any theory of mind at all, and even if they do, whether their model of themselves (i.e. their own "mental states") is sophisticated enough to make such a prediction.

Even humans aren't that great at predicting how they would have acted under slightly different premises! Why should LLMs fare much better?