Hacker News new | ask | show | jobs
by littlestymaar 638 days ago
GP is talking about context windows, not the number of token used by the tokenizer.
1 comments
sva_ 638 days ago
Somewhat confusingly, it appears the tokenizer vocabulary as well as the context length are both 128k tokens!
link
littlestymaar 637 days ago
Yup, that's why I wanted to clarify things.
link