Hacker News new | ask | show | jobs
by kaoD 306 days ago
> the LLM perceives [...] the word as a single indivisible thing (a token).

Two actually, "blue" and "berry". https://platform.openai.com/tokenizer

"b l u e b e r r y" is 9 tokens though, and it still failed miserably.