[PeterisP]

The key difference is what I'd call "context-free embeddings" vs "contextual embeddings". Due to its structure, word2vec and similar solutions have to assign every single "bank" in every sentence the exact same vector, but later models (e.g. all the transformer models, BERT, GPT, etc) will assign wildly different vectors to "bank" depending on the context of surrounding words for that particular mention of "bank".