[krackers]

I had hoped the OP article would have gone into more depth on the intuition as to why you'd expect deltanets to work at all. It seems like we're going back to LSTMs and RNNs where you compress the history back into a fixed-size hidden state. Aside from the easy parallelization in training, I thought that Attention worked much better because it got over this fundamental bottleneck and just let every token interact with every past token (of course you pay for it in compute, but ultimately that's what got us a GPT).

I guess that's probably why you still need some MLA layers in there.