Y
Hacker News
new
|
ask
|
show
|
jobs
by
charlieyu1
272 days ago
I don’t think this can be solved until there is massive investment to train LLM in native Japanese. The current ChatGPT tokenizer still use BPE and you can’t even present a Japanese character with a single token