[not-a-llm]

you use the wrong word

the Anthropic tokenizer is not worse, its more expensive/verbose

[nullsanity]

So, worse? Because we benchmark off token use when talking about token use, and everyone else understood that.

[marcosdumay]

The most important feature of a tokenizer is dividing the inputs into independent values that the neural network can work with. It's not the size.

[jascha_eng]

I mean it might lead to better performance on the model side. So the tokenizer is better but more expensive.

[otikik]

It’s better for them