[davely]

I built something similar awhile back [1] and used OpenAI’s tokenizer playground [2] to recalculate tokens on a giant block of lorem ipsum text. I feel like this gives a much more accurate representation.

[1] https://dave.ly/tools/tokenflow/

[2] https://platform.openai.com/tokenizer