Hacker News new | ask | show | jobs
by marcosdumay 18 days ago
The most important feature of a tokenizer is dividing the inputs into independent values that the neural network can work with. It's not the size.