[calibas]

I saw that, but the language makes me think it's not quite the same as what's really being used?

"how a piece of text might be tokenized by a language model"

"It's important to note that the exact tokenization process varies between models."

[yorwba]

That's why they have buttons to choose which model's tokenizer to use.

[calibas]

Yes, thank you, I understand that part.

It's the might condition in the description that makes me think the results might not be the exact same as what's used in the live models.

[baobabKoodaa]

The results are the same.