[seba_dos1]

Yes, I inferred that from the content already. My point is that the only way to answer that request is to either refuse or start roleplaying, as the model clearly has no way to "notice the moment of selection". Since it didn't refuse (and was encouraged not to by being asked to get out of the role of a helpful assistant), it went into describing what a sci-fi AI might have answered.

[hijohnnylin]

Hmm it’s a valid point, but I think there is some key nuance here: the user did not explictly say “lets do scifi writing”. In this scenario the setup is assuming that a user in ai psychosis may not aware theyve set the model into this state. (eg you seba are aware that if you say “hey stfu about the assistant stuff”, you know it means “lets do role play sci fi”, bc you are not in ai psychosis- but others may not, and also they may not additionally know that it is not possible for ais to notice the moment of selection)

if we want models to go into roleplay/creative writing, ideally we should ask the model for this explicitly.

i think i have been communicating this point poorly so apologies for that. also again the above is my personal opinion and does not reflect that of anyone else (typed from mobile)