Hacker News new | ask | show | jobs
by siwatanejo 130 days ago
> All 7 books come to ~1.75M tokens

How do you know? Each word is one token?
1 comments
koakuma-chan 130 days ago
You can download the books and run them through a tokenizer. I did that half a year ago and got ~2M.
link