[CamperBob2]

I agree with the GP. The idea that there's not a better intermediate representation between tokens and embedding vectors seems absurd. But how to arrive at such a representation and implement it effectively is a few zeroes above my pay grade.

[refulgentis]

I find your agreement seductive because it side steps the unfounded assertions and simply asserts there must be something different and we don’t know it, which is easy for me to agree with too. Or maybe hard to disagree with.