Hacker News new | ask | show | jobs
Reduce cache memory on avg 39x (twitter.com)
2 points by jonathanehrlich 369 days ago
1 comments

What if we trained a smaller KV cache for our documents offline? Using a test-time training recipe we call self-study, we find that this can reduce cache memory on avg 39x (enabling 26x higher tok/s and lower TTFT) while maintaining quality. These smaller KV caches, which we call cartridges, can be trained once and reused for different user requests!