[mhartz]

Can someone help me understand Figure 2? Why does the newest token appear at the beginning of the sequence rather than next to its neighboring token?

[nivekkevin]

it's a rolling buffer, so it just upsert index % 4 in this case

[mhartz]

Thanks, so does that mean position within the buffer is irrelevant?

[nivekkevin]

it does feel like so, the position eventually loses its meaning as more and more data gets crunched by the training process, eventually it's just a context of the past 4 tokens it feels like