|
|
|
|
|
|
by ubedan
6 days ago
|
|
|
Spectacular... Reminds me of the SimdJson algorithm in terms of jaw dropping nearly unbelievable speeds through creative programming. I hope this code get popular, as it will save tons of electricity, money, CO2, etc. Have you considered publishing a rust crate as well? (If not, I volunteer.) |
|
|
I don't think tokenization has ever been a meaningful bottleneck. JSON being fast falls into the same bucket much of the time. We spend way more energy on I/O and storage than we do on serialization and tokenization.
If you are concerned with economics and the environment, request batching would make a bigger impact. The most expensive part of this whole thing is GPU underutilization. You can save 50% with OAI right now if you can figure out how to make your workload fit the batch pattern. Do your users always need answers right now or can we afford to wait a few days in some cases? Tool calling doesn't "time out". Wall clock does not exist in the LLM. It took me a while to get used to this.