Hacker News new | ask | show | jobs
by kouteiheika 3 days ago
The model is natively quantized (i.e. it was trained that way in the first place, so this is not a post-training quantization which degrades performance).
2 comments

Isn't it not completely quantized? I thought there were some dense parts but most is int4?
Often in MoE models the experts are quantized while the shared portions, being a much smaller part of the network with greater impact, are kept at higher or full precision. Not familiar with the Kimi QAT approach specifically but it's likely they do this.
But the huggingface link mentions BF16, F16, and I32?
Not every weight is quantized. For example, those weights which don't take much space or are highly important are left in higher precision. State-of-art quantization of weights is never done uniformly (i.e. to all weights and in the same way).
I don't believe safetensors has a native int4 dtype, so they packed 4 int4s into a bf16 in this checkpoint.