[woadwarrior01]

dot-product attention is the biggest barrier. This is why there are so many attempts to linearize it.

[amitport]

that fail... linearization is a bad idea. But plenty of other optimizations are done