Hacker News new | ask | show | jobs
by abofh 1 day ago
If you just want a smaller vocabulary, use French? If your goal is to communicate to an LLM, maybe saving tokens isn't the all in win, unless you like reading assert gronkHitThing(true)
1 comments

French text is somewhere around 10-30% longer than the corresponding English text. I would guess much of what you save on smaller vocabulary is lost on the lengthened text.
What would matter more is the token length, depends how well your tokenizer was trained on french I guess.