[Aurornis]

A dual Xeon of this era is probably pulling 300W or more when loaded.

At national average electricity prices, that’s $1.35 per day. More during the summer if you have to cool the space.

If you run it 24/7 and ignore prompt processing time (not a good assumption at all) it would get around 400,000 tokens in a day.

That’s about $0.30 per million output tokens.

Coincidentally, that’s the same price for this model on OpenRouter right now, but OpenRouter token gen will be 8X faster.

There are a lot of good reasons to experiment with running LLMs locally, like if you don’t want any data leaving your house.

Don’t think that you’re going to come out ahead monetarily. I say this as someone with a lot more money invested in local inference hardware at home. It’s fun, but it’s not a way to save money.

[brailsafe]

Reasonable analysis, especially because this person seems to have an actual house. In my case, I rent and don't pay for electricity directly, so the cost effectiveness threshold is whenever the landlord starts complaining

[twsted]

I think, may be actually wrong, that most of us do not consider running a model locally a way to save money. It is a way not to spread personal info around.

[Aurornis]

Anyone running LLMs at home will come to that realization quickly, if they’re looking at their power bills. Even feeling the heat output of a computer running at 100% in your office makes it clear.

I was responding to a lot of the comments saying this was a reasonable way to avoid paying for tokens or subscriptions. I don’t want anyone getting the wrong idea that this is a way to save money if that’s their priority.

[euio757]

> Even feeling the heat output of a computer running at 100% in your office makes it clear.

What does it make clear? That I can replace the space heater my wife runs 9 out of 12 months of the year with a home server? And effectively get $0.00 per token during those times?

In houses running A/C year round, sure there'd be some impact, but in all the places running heat, doesn't seem that it'd move the needle on power bills.

There are startups whose entire business model is "cloud server as a home space heater" (aka "data furnace") ...

[Aurornis]

> What does it make clear?

That it’s pulling a lot of watts.

Good for you if it’s replacing electric space heaters.

[Loocid]

There are bunch of bitcoin miner products now that are packaged to look like regular space heaters.

[pjmlp]

I never get this kind of discussion on HN, many of us live in regions where this is a non issue.

[hahahaa]

It is like crypto currency - for most people more cost effective to buy than mine due to power prices.

[bitexploder]

It gets better in the cooler months when heating is running in a home :)

[hahahaa]

Sort of - heat pump could be 2x to 3x more efficient. Heat pumps cool outside air down to get some of the energy for heating inside.