[Ilaurens]

"Interestingly, Kimi K3 got rid of all RoPE layers and uses NoPE (No Positional Embeddings) everywhere instead."

It just baffles me that this even works at all. Doesn't it just become a token soup? Is attention that precise that a second token can tell its the second token just because it learns to accumulate something in the embedding space without any sort of inductive bias?

[yhocd]

The primary author of RoPE, Jianlin Su, is currently working at Moonshot (KIMI)

[itkovian_]

Causal masking allow model to learn implicit positional embeddings. The meme that a transformer block is permutation invariant is not true.

[ModelForge]

And adding to that, there is also the recurrent state in the Kimi Delta Attention. I wouldn't call it position information but more sth like "position sensitivity"

[dwohnitmok]

As a sibling comment points out you don't strictly need positional embeddings for decoder-only causal transformers. You definitely need it for non-causal ones (e.g. the encoder of the original transformer paper!).

And yes accumulation is a good intuition for what's going on. You could imagine a part of the attention head that just kept writing to the same part of the residual stream causing that to keep accumulating (simply via attention summation) as more input tokens come in thereby functioning as a kind of index without the need for any positional encoding.

[augment_me]

When you have recurrent blocks in your model, you implicitly have a timestep T(amount of recurrent steps). Similar to Diffusion Transformers, it then becomes valuable to encode the knowledge of where you are in this chain somehow. NoPE is more flexible than RoPE for this.

[fspeech]

Linear layers use decays (like IIR filters) that naturally provide relative positions. Full attention layers can then be free to develop concepts that attend to each other regardless of distance.

[natrys]

At least the SSM layers should have relative PE built in via recurrence, in a hybrid attention model like this.