[montebicyclelo]

It wasn't obvious that a transformer could do this, and learn to produce conv via attention

[artemisart]

But it is, as long as the positional embedding are sufficient, i.e. use relative positional embeddings here.