[thomasahle]

> by injecting inductive bias like positional encoding

I don't think it's fair to just call positional encoding "inductive bias". The positional encoding is the only way the word order is communicated to the model. That would be like saying it is inductive bias to include color channels when working with images.