Hacker News new | ask | show | jobs
by zkmon 6 days ago
How can the smaller Unsloth GGUF quant can beat the original google quant? (ref: unsloth/gemma-4-31B-it-qat-GGUF)
1 comments

I may be wrong, but this is what I figured out. Google provided these quantize-ready models, but they do not come pre-quantized. However, to produce their benchmarks, they quantized their model using the standard quantization approach. Unsloth has an advanced quantization method that performs better than the standard quantization, so the evals are better for unsloth quants.