[igorkraw]

Your last point is true only with positional encodings though, attention itself is a permutation equivariant function