Y
Hacker News
new
|
ask
|
show
|
jobs
by
beering
12 days ago
The set of tokens is learned, more or less. So I don’t get what point you’re trying to make here. There’s not a human manually deciding what tokens make up the token dictionary.