Hacker News new | ask | show | jobs
by mnks 1280 days ago
Have a look at https://beta.openai.com/tokenizer which uses javascript reimplementation of the GPT-2 / GPT-3 BPE tokenizer. In this case it's [31373, 995].