Indeed it would! Is anyone here going to try to do that?
As an observer is needed to assess the LLM, perhaps the easiest test is copy-paste between two instances and then ask chatGPT, or whichever LLM, "who were you talking to?".
You can’t use two instances. They both would have individual selfs.
I think an experiment would be to feed back whatever a LLM says to that same LLM, and see whether they’ll, at some time, say “why are you doing that to me?”
I tried with a few variations, GPT 3.5 and 4 seem to be pretty aligned in not expressing themselves when not asked a question. "Our conversation seems to be in a loop, if you have anything I can help you with ..." blah