[m3at]

Two of the most popular libraries for token creation (tokenization) are in fact in rust, with an interface in python:

https://github.com/huggingface/tokenizers

https://github.com/openai/tiktoken