Hacker News new | ask | show | jobs
by beering 12 days ago
The set of tokens is learned, more or less. So I don’t get what point you’re trying to make here. There’s not a human manually deciding what tokens make up the token dictionary.