Hacker News new | ask | show | jobs
by jfoster 12 days ago
Could multiple Chinese characters be counted as a single token?
1 comments

Possibly. Telephone (电话) is electricity/electronic (电) + talking/speech (话).

In Japanese there's the Japanese possessive ('no') which can also be a modifier/qualifier in text like 男の子 (boy, literally "man of child") and 女の子 (girl, literally "woman of child"), so there are sequences of Chinese characters (possibly in combination with Japanese) that could be a single token like character sequences in the Latin script.

I've found https://digitalorientalist.com/2025/02/04/to-merge-or-not-to... with some information/analysis of this.