[2001zhaozhao]

This is 128B dense though. the K/V cache on long context is going to be massive

[Havoc]

Don’t think kv size correlates to dense/moe

[zozbot234]

KV size correlates with attention parameters which are a subset of active parameters. So a typical MoE model will have way lower KV size than a dense model of equal total parameter count.

[syntaxing]

With turbo quant, you would reduce it by over 6X.