Hacker News new | ask | show | jobs
by qsera 114 days ago
That is exactly how they work.
1 comments
astrange 114 days ago
No, a token is not a word.
link
qsera 114 days ago
I mean, it is some text.
link
astrange 114 days ago
How do you get from a piece of text smaller than a word to an entire coherent sentence?
link