[dwohnitmok]

As a sibling comment points out you don't strictly need positional embeddings for decoder-only causal transformers. You definitely need it for non-causal ones (e.g. the encoder of the original transformer paper!).

And yes accumulation is a good intuition for what's going on. You could imagine a part of the attention head that just kept writing to the same part of the residual stream causing that to keep accumulating (simply via attention summation) as more input tokens come in thereby functioning as a kind of index without the need for any positional encoding.