[semiquaver]

I’ve been struggling to understand the reason for the newer apparently less efficient Anthropic token encoding. If all inputs are less efficient in this encoding, why does it exist? Has Anthropic released any information that would convincingly show it was anything other than a stealth price hike? Please don’t respond if you are speculating.

[remus]

> Please don’t respond if you are speculating.

I doubt you are going to get a response from an anthropic employee, but I think it is safe to assume they have swapped to a new tokenizer because it improves the performance of their models.

[re-thc]

> the reason for the newer apparently less efficient Anthropic token encoding

Less efficient in token usage but per the blogs; it enables the model to perform better.

[mnicky]

It's really simple I think.

More tokens per same text length means more capacity to encode information. More information means model can potentially perform better.

They introduced it around the time the Mythos came so my speculation is that if you have more capable model at some level you may find the current information encoding not using its full potential.

We will see whether OpenAI also introduces new tokenizer when they come to Mythos-size models.