Hacker News new | ask | show | jobs
by Mindless2112 304 days ago
Tokenization is an inherent weakness of current LLM design, so it makes sense to compensate for it. Hopefully some day tokenization will no longer be necessary.