Y
Hacker News
new
|
ask
|
show
|
jobs
by
sp332
932 days ago
If you put it into the tokenizer
https://platform.openai.com/tokenizer
you can see that it helps in some places but not in others. It pulled out "SEA/OF/TRAN/QU/ILITY", but I think it broke up every instance of the word "THE".