[michaelmior]

This depends on what your workflow is. There are use cases for tokenization that don't always involve immediately feeding the text into a model.

[casey2]

There are more usecases, for this class of tokenizer, now that it's 1000x faster as well. RAG being one example.