[Imnimo]

Yeah, I didn't realize there was that additional prompt. It certainly makes more sense that if the prompt includes the description that the agent is playing the role of an AI, it would be able to deduce that it would not be affected. I was assuming there was no such indication, and so the system would be implicitly trying to predict what a human would say in that situation (since the training data is largely human-written text).

Still, you could imagine a parallel version of Flamingo which performs the same reasoning, but is artificially slowed when shown Stroop images. Obviously, there would be no way for it to deduce this fact from the training data, and it would not be able to say that it is also affected as humans are.

I don't mean to say that this is some great failing of the system or anything - just that a casual reader might infer from the Stroop dialogue that the system had some way to inspect and reason about its own performance, when in fact it's just estimating what it thinks would be true for AI systems (since it was told that it's an AI system in the prompt) in general based on the training corpus.