[kpw94]

In the context of local LLMs on limited hardware I've ran to the exact same conclusion: "tok/s" isn't the most useful metric when my personal North star metric, given my fixed hardware is: Model smart enough to execute my goals _in the minimum amount of time_.

Some models I tried (Mistral I think) had better tok/s, and roughly same billion parameters / scores on various benchmark... But they were _so_ verbose, that they generated many more tokens compared to a Qwen model of same caliber to answer the same thing.

So even though it had better generated tok/s, because so many more were generated, the clock time was longer.

And this compounds over mutli-turns: more generated token means more context used in the next turn (until some compaction or something runs)

[c7b]

Even more important in a local context is the difference between token generation and prompt processing speed. We tend to focus on the former, but for multi-turn/agentic workflows the latter can dominate.

[kpw94]

Yeah definitely. I've recently commented on that: https://news.ycombinator.com/item?id=48557890

[PunchyHamster]

I feel like we need to see more proliferation of local LLMs to start seeing ones turned to be terse, rather than maxing the amount of tokens user pays for