Y
Hacker News
new
|
ask
|
show
|
jobs
by
mnks
1280 days ago
Have a look at
https://beta.openai.com/tokenizer
which uses javascript reimplementation of the GPT-2 / GPT-3 BPE tokenizer. In this case it's [31373, 995].