Y
Hacker News
new
|
ask
|
show
|
jobs
by
kardos
1 day ago
French text is somewhere around 10-30% longer than the corresponding English text. I would guess much of what you save on smaller vocabulary is lost on the lengthened text.
1 comments
ViscountPenguin
1 day ago
What would matter more is the token length, depends how well your tokenizer was trained on french I guess.
link