[koheripbal]

I honestly don't understand why they don't just set words as tokens. Is the dictionary really that big?

[cyclotron3k]

Yes, it really is! Because you're not actually dealing with a dictionary of English words - you're dealing with all possible combinations of characters, from all unicode character sets.

Here's a really good explanation: https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...

[qup]

Your question belies the truth in your statement.