Y
Hacker News
new
|
ask
|
show
|
jobs
by
staticautomatic
522 days ago
They’re called “lemma”
1 comments
Arnavion
522 days ago
The intent is the same, but as I understand it LLMs don't tokenize based on lemmas, though some of the tokens probably line up with them.
link