[simonw]

I hadn't realized OpenAI's tiktoken Python library could work with other models outside of the OpenAI family, that's really useful: https://github.com/therealoliver/Deepdive-llama3-from-scratc...

[moffkalast]

It's more than just that, practically every notable open model released in the past year or so uses tiktoken as the tokenizer.

[therealoliver]

I'm glad to have helped you :)