[marzukia]

The most counter-intuitive bug was that a unique message ID in the system prompt broke the entire KV cache. Since the cache requires byte-exact matches, that changing ID forced a full re-compute on every turn, turning warm contexts into cold fills.

I've open-sourced the fork (qMLX) and a benchmark script (bench_qmlx.py) that separates prefill/decode metrics. I chose to fork rather than submit a PR because these hybrid attention changes are specific to the Qwen flavor of models and would likely be unpalatable to upstream maintainers who prioritize a general-purpose stack. I expect this fork to continue diverging from the base as we optimize specifically for this architecture. Happy to answer questions about the caching strategy or eviction logic.

[msdz]

> a unique message ID in the system prompt broke the entire KV cache. Since the cache requires byte-exact matches, that changing ID forced a full re-compute on every turn, turning warm contexts into cold fills.

This is (part of) the same problem that initially lead Anthropic to ban non-Claude Code clients from using the subsidized subscription: A full to-the-second datetime stamp in the system prompts of OpenCode, and I believe Pi as well, invalidated the caches, making this a very expensive use of their compute very quickly.

They even had Anthropic employees submit PRs (or maybe just open issues, I’d have to check) to these other clients/harnesses because the cache misses were hitting them so hard.

[djmips]

fascinating! That's a useful thing to know.

[supermatt]

> The most counter-intuitive bug was that a unique message ID in the system prompt broke the entire KV cache.

How is that counter-intuitive? You changed the key, so it didn't return a value. Thats a KV cache working as intended.