[joewferrara]

They show that a decoder only transformer (which gpts are) are rnns with infinite hidden state size. Infinite hidden state size is a pretty strong thing! Sounds interesting to me.

[visarga]

not infinite, just scaling linearly with length