[cowwoc2020]

Makes me wish that shortly before the server-side expiration, we could save the cache on the client-side, indefinitely.

But my understanding is that we're talking about ~60GB of data per session, so it sounds unrealistic to do...

[porridgeraisin]

I wanted this as well. Even asked about it at an openai talk. Basically a way to get the KV cache to the client (they can encrypt it if they care about me REing it, make a compressed latent if they don't wanna egress 20GB, whatever, I'm fine with a black box) so that I can load it later and avoid these cache misses.

I think the primary reason they cannot do this is that they change the memory and communication layouts in their serving stack rather aggressively. And naturally keeping the KV cache portable across all such layouts is a very difficult task. So you'd have to version the cache down to a specific deployment, and invalidate it the moment anything even small changes. So giving the user a handle to the cache sort of prevents you from making large changes to memory layout. Which is I suppose not that enticing. Also, client side KV caches are only meaningful in today's 1M contexts. Few y back it wasn't necessary, since just recomputing would be better for everybody.

To be clear, I don't mean they send it along with every request. Rather, they do their current TTL cache, and then when I'm at the end of a session, I request it in one shot and then close the session. And it doesn't have to come to the literal client, they can egress it to a storage service that we pay for, whatever. But ya the compat problem makes it all a non starter.

[yumraj]

Where are you getting 60GB from? It shouldn’t be that large.

But yes, would love to save context/cache such that it can be played back/referred to if needed.

/compact is a little black box that I just have to trust that is keeping the important bits.

[davmre]

The KV cache consists of activation vectors for every attention head at every layer of the model for every token, so it gets quite large. ChatGPT also estimates 60-100GB for full token context of an Opus-sized model:

https://chatgpt.com/share/69dc5030-268c-83e8-92c2-6cef962dc5...

[CraigRood]

That is actually nuts.... I'm trying to understand the true costs of AI, wonder how I plug this in!

[visarga]

There are ways to quantize or compress KV cache down.