Hacker News new | ask | show | jobs
by minimaxir 1281 days ago
So it does.

The code there implies cl100k_base has a vocab size of 100k (I guess it's in the name lol) which means it is more comprehensive than GPT-2's 50k, so fewer tokens will be necessary.