|
Hi HN. Follow-up to the Xeon post from a couple of weeks ago. A lot of people came away from that one with a 25-flag command and no real idea which flags materially did anything, and the honest truth is neither did I fully. So I went back and measured it, one flag off at a time, 174 server restarts, with the engine log as the source of truth instead of my own assumptions. This also is an attempt at starting to answer the "cool, but do you have numers for this" question, which is harder than one would assume! Two things I'll flag up front, because they're corrections to my own post: --spec-autotune, which I'd called the way to squeeze the most out of speculation, is actually the worst speculation setting I tested. Ouch. And --mla-use, which I presented as active, isn't even wired for Gemma 4 and gets silently ignored. More broadly, most of the config does nothing for a typical setup if you're not "holding it right". The flags that genuinely carry it (flash attention, the physical-core thread count, a fixed draft length) are a much shorter list than the command suggests, and the drafter turns out to be a win for code but a loss for summarization. And to be clear. The numbers are specific to this box, and none of it runs without the ik_llama.cpp fork I link. I'm still not an ML engineer, so if I've gotten something wrong I'd honestly rather hear it, that's the best kind of reply. The box is now back to being busy as a Nix cache once again now, so answers are best effort. (There's a... fifth(?!) post coming on benchmarking quantization, where the single fastest config I measured turned out to be pure garbage. But making it not garbage was useful. No spoilers!) |