Hacker News new | ask | show | jobs
by charlieyu1 272 days ago
I don’t think this can be solved until there is massive investment to train LLM in native Japanese. The current ChatGPT tokenizer still use BPE and you can’t even present a Japanese character with a single token