[Nopoint2]

I don't get the point. A simple CNN with stride =1 should be able to solve it perfectly and generalize it to any size.

[montebicyclelo]

It wasn't obvious that a transformer could do this, and learn to produce conv via attention

[artemisart]

But it is, as long as the positional embedding are sufficient, i.e. use relative positional embeddings here.