[eggdaft]

The context comes from the attention mechanism, not from word embeddings.

[mjburgess]

Run attention on an ordinal word embedding and see what happens

[eggdaft]

Well yes, necessary but not sufficient, obviously.