Hacker News new | ask | show | jobs
by Aurornis 84 days ago
I should clarify that I'm referring generically to the types of quantizations used in local LLM inference, including those from Unsloth.

Nobody actually quantizes every layer to Q4 in a Q4 quant.