[amkkma]

Is there any hope of understanding this with just calc and linalg knowledge?

[adamnemecek]

I think so. The main idea is the idea of Hopf coherence. The transformer/Hopf algebra update their internal state in order to enforce the Hopf coherence formula (you can find that in the paper).

The idea of streams (as in infinite lists) is related to this via coalgebras.