[mp187]

Why was this your first thought? Is a limiting factor to transformers the MLP layer? I thought the bottleneck was in the renormalization part.

[brrrrrm]

At small input size, yes the MLP dominates compute. At large input attention matters more