Hacker News new | ask | show | jobs
by raverbashing 916 days ago
Do they all use the same tokenizer? (I mean, Mistral vs GPT)
1 comments
superkuh 916 days ago
No. Mistral uses sentencepiece and the GPT use tiktoken.
link