Hacker News new | ask | show | jobs
by Kalabasa 996 days ago
How does that fare for non-English queries? E.g. are two chars still not enough for Chinese languages?
1 comments

I've seen 1.5 chars/token used as a rule of thumb for estimating token counts in Chinese text.