[whimsicalism]

transformers were also just better at the LM task than 2018 RNNs for equal amount of flop training

[VHRanger]

Yeah, that's just the training stability part to my knowledge

[whimsicalism]

they're also just less capable models. like just adding attention on top of an RNN made them a lot better

[SpaceManNabs]

Calculating self-attention is still quadratic though. So you get the negatives of transformers there too.