[Wowfunhappy]

If you were iterating on code with a human partner, how fast would they write?

Like, obviously faster is better and the entire point of computers is to do things faster, but I find it kind of surprising how many people consider 9 tokens per second—heck, even much less than that—to be unusably slow. It's still automating a class of task that virtually no one before 2022 was able to automate, and it's faster than basically any human can write code.

I guess the better question is, why would you use a 9 token per second system when you could use a much faster cloud model. Obviously, if you want speed, that's the way to go. But a lot of people seem to find the idea of sending their requests to a third party server untenable. If 9 tokens per second is the best you can do—I don't know, that seems usable and useful to me.

[mh-]

This isn't just measuring the speed of tokens that are written to a file (code). Your human partner is thinking a lot faster than 5 tok/s.

[sigmoid10]

I guess you haven't used Codex/Copilot much? All good LLMs now use reasoning. Especially for code it is almost a requirement to produce correct results. But that means even having them implement a comparatively simple function or algorithm can easily produce 3000 output tokens. Waiting 5+ minutes every time you iterate because you want a tiny tweak in the code is not a practical strategy for live development.

[imtringued]

You have to remember that the CoT thinking tokens take a while to produce. If your model produces 900 thinking tokens you're waiting almost two minutes for the first token.

The biggest argument in favor of running local models doesn't seem to be privacy at all, it's the fact that you can't run out of tokens even if it is a bit slow.