[great_psy]

The part that is surprising, is where the reverse does not have a higher than random probability.

Sure, most of the time “is” is not commutative, but sometimes it is, so it should be higher probability than guessing.

[phagenlocher]

I don't find that surprising, as LLMs typically predict the next token in a string of tokens not the previous. Just because we, as humans, have the knowledge that "is" can be a symmetric relation does not imply that LLMs have this knowledge.