[Ambix]

For those who interested, there some new researches in the field [0]. It usually possible to create more compact token representation from given text, but my guess the greedy "optimal" tokenizer might harm the performance of the model?

[0] https://www.reddit.com/r/LocalLLaMA/comments/140gcn7/new_tok...