[nisegami]

How consistent is the tokenization across different model families? It always served as a mental hangup for me when comparing LLM inference pricing.

[alexellman]

They all tokenize a little differently so they are not exactly 1-1. However I plan on addressing this by having each model complete a test task and getting the actual price from each api + token count to make a real 1-1 comparison.

[esafak]

And please timestamp the benchmarks, and rerun them periodically, so vendors can't quietly cost optimize the model when no-one's looking.

[nisegami]

Ah, that's a great idea and would be a welcome addition to the site.