Hacker News new | ask | show | jobs
by kwrobel 2396 days ago
I agree. Perplexities (probability of a text) can be compared using different tokenization after normalization.