[elcritch]

What are other barriers in transformers? Or is the normalization layer the primary one?

[woadwarrior01]

dot-product attention is the biggest barrier. This is why there are so many attempts to linearize it.

[amitport]

that fail... linearization is a bad idea. But plenty of other optimizations are done