[ec109685]

Caching doesn’t work the way the bug reporter implies. Caches are shared (at least across the enterprise), but its key is always a function of the input before it.

We achieved significant savings simply by moving everything that varies across individuals out of the system prompt so every session starts from a cache point.

For example you never want your system prompt to start with the time that the session started. Move that to the first user message if needed.

[macNchz]

Caching is not supposed to work like that, but that doesn’t preclude the cache key computation function from having bugs.

[marginalia_nu]

Yeah there's quite a lot of potential bugs that could have this shape. If I were to guess it could be a buffer in a buffer pool not being sized and zeroed correctly, allowing stale data to bleed between sessions.

[nok22kon]

or the cache retrieval function for a key retrieving the wrong entry

[Waterluvian]

There is a massive incentive for optimization, so I expect they’re doing a ton of very clever tricks, all of which make this kind of bug more likely.

[estebarb]

Hash functions necesarily have collisions. Also, it is perfectly possible to introduce bugs in the hash function (hash inputs, hash function itself) that allows cross account contamination.

[margalabargala]

Hash functions necessarily have collisions, but it's perfectly possible to make the expected time between collisions greater than the human lifespan.

[supriyo-biswas]

There could just also be a bug where the output tokens of session 1 were shared with session 2, due to a race condition or similar.