[foxhop]

The llama 3.0, 3.1, & 3.2 all use the TikToken tokenizer which is the open source openai tokenizer.

[littlestymaar]

GP is talking about context windows, not the number of token used by the tokenizer.

[sva_]

Somewhat confusingly, it appears the tokenizer vocabulary as well as the context length are both 128k tokens!

[littlestymaar]

Yup, that's why I wanted to clarify things.