Hacker News new | ask | show | jobs
by axiom92 1500 days ago
Yeah it's BPE. OpenAI has a nice tool that allows you to play with the tokenizer https://beta.openai.com/tokenizer.