[p0]

How does this compare to the BPE crate [1]? Its main selling point is support for incrementally re-tokenising text, but it's also faster than tiktoken.

[1] https://crates.io/crates/bpe

[matthewolfe]

I'm working on incremental re-tokenizing next. Then I'll run some benchmarks against this crate too.