[canjobear]

The pretraining approach was used in vision for years before it was successful in NLP.

[julien_c]

Not really on unsupervised/self-supervised data though, right?

(nor on the same scale of corpora, as far as I can tell)