[beacon294]

I find that fp8 cache can be pretty bad in vllm but works fine in llama.cpp. I don't know why, but I plan to review the implementations.

[CMay]

Llama.cpp implemented some rotation optimizations for quantized kv cache to improve the preservation of attention quality or similar, after everyone was talking about TurboQuant. It's not perfect and when you're talking about long form reasoning, little differences can make or break the results so it is situational.

[beacon294]

I'll read it. It could be the quants too. Some quants I try are inexplicably bad, some seem better than official (or unsloth) quants... even what should be run of the mill gguf quantization.