[pamelafox]

Just curious whether it's possible to push any of your performance improvements to tiktoken itself?

[matthewolfe]

I probably will. Was hesitant initially, because adding PCRE2 as a dependency might cause issues to existing projects. I believe this was discussed briefly in a closed PR with other performance improvements.