[svara]

> Why WOULD it be assumed to be possible to get some solid generalized output via next token prediction when we haven’t seen it yet?

Because it's such a general concept that it doesn't imply any important limits in and of itself, as far as text based AI goes.

It really just means creating an output sequence from an input sequence in a discrete, iterative manner, by feeding the output back into the input.

Regarding your example, I've got to admit that's hilarious. I'm not sure it's as much of a fundamental issue even with current state of the art models that you make it sound; rather they're trained on being usable for role play scenarios. Claude even acknowledged as much when I just tried that and lead with "In this imaginative scenario, ..." And then went on similarly to yours.