[mikewarot]

I strongly suspect it's tokenization that drives this. If we trained with ASCII or even UTF8, I think we'd have much better results.

[bellowsgulch]

Self-attention is an O(n^2) operation. You don't want to train on individual characters, or bytes.