[hagen8]

Some ppl don't like to hear it. But I would assume that token costs when using an inference provider are cheaper than electricity of using locally.

If we just take into account output token generation for simplicity. With 5tps u get 18k tokens an hour. That would costs around 0.005USD from an inference provider.

I estimate that the server consumes probably around 500W during inference.

In Germany where 1kwh cost around 0.3USD, 18k tokens inferred locally would therefore cost 0.15USD which is 30x the costs of using an inference provider.

But for ppl who worry about their data, running locally might still be good. However, they should be aware, that it is much less efficient than using an inference provider.

The efficiency gap will also significantly increase as new GPUs will make inference much more efficient.

EDIT: I first thought it'd be 180k token, but thanks to someone mentioning in the comments, it is 18k. I guess with that, it will be tough unless u got electricity almost for free. Also, the inference providers are probably still using H200/H100 for those small models. Once they use GB300 or next year the new Ruby GPUs, inference will be cheaper by a factor of 30. By then, running local models will mostly be about privacy.

[throwawayffffas]

I run qwen 27b at home when working it pulls around 400W. I get 40ish tokens per second generation and more importantly about 1000 tokens per second prompt processing.

In an hour it can process 3.6 million tokens or generate 144000 tokens. This costs me about 15 cents given my electricity prices.

For sonnet the equivalent token costs are 7.2 dollars for the prompt processing or 1.4 dollars for the generation. The cloud is 10x more expensive for generation and close to 50 times more expensive for processing.

[why_only_15]

The correct comparison is not sonnet, but qwen3.5-27b on a cloud. Alibaba's pricing [0] is $0.20/m input $1.56/m output, so $0.72 for the prompt processing or $0.22 for the generation. Yours is still cheaper but the margins are less.

My guess is that this math gets less good with MoE (because you will be limited by VRAM, but clouds won't).

[0]: https://openrouter.ai/qwen/qwen3.5-27b

[throwawayffffas]

I run 3.6 but yeah you are right

[Figs]

> it pulls around 400W

Try dropping the power cap on your GPU if it supports it; you can often get much lower energy usage with minimal loss of tok/s (particularly during generation) than whatever the GPU defaults to. There's a sweet spot around 200W on the GPU I'm currently testing that gives me about ~75% of the max pp and 97% of tg while using 100W less than the default/max 300W power cap -- and the card runs much quieter as a result.

[bischofs]

I don't pay anywhere near 0.30usd in the US - I pay half that off peak and can buy 1000$ worth of batteries to load up on super off peak (0.11usd). Also the inference providers are fighting over market share with huge debt loads so they are definitely going to go up in price.

[hagen8]

Inference costs will go down massively once they use the upcoming GPUs. I estimated that a model like GLM5.2 will be around 0.03USD/M output tokens in 2 years when the Feynman GPUs will be available in 2028. And this did not even consider architectural efficiency improvements. In mid 2027 we will already see a 10x reduction once everyone has switched to the Ruby architecture.

It will be feasible for everyone to have 20 different agents running at all times. A new world is coming

[amanaplanacanal]

Yeah I had to check, I'm paying 0.08usd per kwh. This is in the US PNW with quite a bit of local hydro power.

[brewtide]

It's all relative. On the opposite coast, Maine it is ~ $0.28 cents kwh including getting it there. (~ 50% energy, 50% delivery). It's too darn expensive here.

[sosodev]

Of course efficiency matters, but a lot of people either have cheap electricity or efficient hardware. My AMD strix halo home server can serve Gemma4-26B at like 70 TPS (rough estimate, I don’t remember the exact speed buts its fast af) while only using 100W.

[segmondy]

don't care, and yeah i don't like to hear it. we don't run local because it's cheaper money wise. we do it for freedom, for privacy and having option makes it cheaper in the long run. if there was no local options, your cloud model would cost much more!

[albrewer]

Not OP, but your math is a bit off - I have solar panels :)

[senordevnyc]

There is something genuinely beautiful to me that these goofy apes have managed to turn sunlight and sand into intelligence.

[bigyabai]

It's the "Race-to-Idle" situation all over again. It consumes less power to complete a task faster, whereas using "low power" hardware that draws max TDP for 30 minutes isn't very power efficient.

The privacy nuts have a better leg to stand on, but even then it's hard to believe that they're using on-prem AI to replace SOTA model inference. As cool as local LLMs are, a lot of the stuff people run is a novelty.

[1d22a]

To this end, I've been thinking that it would be cool to create a solar-battery powered ("off grid") server providing a self-hosted LLM service. Offline when the sun doesn't shine enough (like Low Tech Magazine[1]). At whatever size is required for every day, community-scale use (a friend group, a street, a club). Fix the data centre issue by democratising AI to such an extent that we can bring it into the hands of communities to actually control (and democratically decide their own level of censorship / alignment). Along the lines of some of Geohotz' writing.

Within a short time I think open source models will all be getting good and efficient enough to make it viable to serve this on 2nd hand hardware for cheap. All it will take is a nerd in every small community to pool together a few hundred bucks initial outlay, and then ongoing costs are near free without electricity to pay for.

[1]: https://solar.lowtechmagazine.com/

[dlcarrier]

From what I've seen, most inference providers are running at a loss, so it wouldn't be at all surprising if using their services costs less that running the same software locally.

The commodification of the hardware needed is probably a larger factor, because by the time a baseline computer has enough RAM and processing power to run a desired LLM, that hardware will be efficient enough that the extra electricity usage is nominal.

[kylemaxwell]

You pay 3x as much for electricity as I do, so the math here is going to work out very differently depending on a lot of factors.

[carra]

> Some ppl don't like to hear it. But I would assume that token costs when using an inference provider are cheaper than electricity of using locally.

Maybe, but for how long? Prices keep going up, and every new model eats more and more tokens...

[leobg]

I’m in DE, too. Same calculus. Excluding winter though. When it’s cold outside, a GPU heats my office and I get tokens free on the side. :)

[apitman]

It would be more efficient if you had multiple users (or agents) making parallel requests to take advantage of batching, right?

[Waterluvian]

Locally I’m looking at about CAD $0.05 per kWh when off peak.

[newusertoday]

its 18k not 180k