[lostmsu]

Maybe I am missing something, but I don't see any learning without autodiff.

[adamnemecek]

I thought you were asking about attention only transformers. This paper touches on some of it https://arxiv.org/abs/2212.10559v2.

[lostmsu]

The paper speculates that it is analogous to gradient descent and empirically confirms it is similar in behavior, but it is not a rigorous proof of any kind.

The momentum experiment they made also does not seem related. E.g. it just adds past values to V, which extends the effective context length.

[adamnemecek]

> but it is not a rigorous proof of any kind.

Such is the nature of early theories.