Y
Hacker News
new
|
ask
|
show
|
jobs
by
tecleandor
27 days ago
Tests I've done with NO and FI texts, for the same number of characters, with the GPT5 tokenizer I get around 2x the tokens than EN. With the older tokenizers it's more like 2x or even 3x.