[mikewarot]

If you want an LLM that can do proper spelling, arithmetic, etc. you'll have to train it on ascii or unicode instead of tokens. This would likely increase training times, and possibly also model size, by an order of magnitude or more.

On the plus side it could then correctly count the number of r's in strawberry.

An aside: you wouldn't believe how easy it is to grow strawberries. Mostly it's a matter of containment.