[fooker]

> Even if the output is like 5-6 tok/s, that might be usable for some purposes.

You'll spend ~100x more on electricity than the API cost to have it run on someone else's GPU at several hundred tokens per second.

I think some sort of extreme data privacy requirement is the only situation that justifies this, but the intersection of {needs absolute data privacy, needs to run SOTA model, cannot afford GPUs} is really really narrow. I wouldn't be surprised if this is an empty set.

[walrus01]

There are a number of use cases where sending the contents of your context and prompts (and the resulting output) to a 3rd party service is off the table as an option, and people will compromise speed for data sovereignty. And not everyone's electricity is equally expensive, I pay about $0.075 USD per kWh. It would for example cost me about $48 a month of electricity (not counting cost of cooling) to run a quad socket Dell R940 for a month.

[mdasen]

That's an unusually low electric rate for the US - way below the lowest state average which is Idaho at 12.4 cents. It's certainly possible that you are getting 7.5 cents including delivery, but I've had friends say that they're "getting 13 cents per kWh" here in Massachusetts, but that's just the supply rate and the delivery is another ~18 cents.

There are parts of states like Grant County Washington that have cheap hydro power, but it's very rare for power to be that cheap in the US. Even if this applies to you, it won't apply to the vast majority of people on here who will have electric rates 2-4x higher.

Average electric rates by region:

    New England            28.1 cents
    Mid Atlantic           25.1 cents
    East North Central     20.8 cents
    West North Central     14.8 cents
    South Atlantic         16.1 cents
    East South Central     15.5 cents
    Mountain               14.6 cents
    Pacific Contiguous     26.1 cents
    Pacific Noncontiguous  42.1 cents

https://www.eia.gov/electricity/monthly/epm_table_grapher.ph...

[zdragnar]

I'm actually getting 11 cents in winter, 13 in summer, but my utility company is a co-op. Average for my state is I think 19 cents.

I think you can get down to around 8 if you are signed up for an interruptible load, or a dedicated off peak load, depending on the company, but yeah, standard rates aren't that low.

[murderfs]

> Pacific Contiguous 26.1 cents

This is a bit misleading, because it's combining the 50 cents/kWh from California with 15ish cents/kWh in Oregon and Washington. Seattle City Light, for example, charges 13.38 cents/kWh on flat rate pricing, and far less with time-of-use billing (8 cents/kWh on off-peak).

[radium3d]

If you run off solar with battery backup, you can achieve lower than those rates! Look at Time of Use rates. The super off peak rates instantly become the max price point once you pair TOU with Solar + battery.

[rootusrootus]

A lot of people quoting low rates are also just referring to their off-peak rate. This is pretty common in EV discussions. It's not exactly a fair argument there, either, because the flip side of having an off-peak rate is that the on-peak rate is usually quite a lot higher. So the true effective rate is a bit higher, somewhere in the middle depending on usage pattern.

[searealist]

It's easy to have your EV only charge off-peak, though. It's just a setting.

[rootusrootus]

My point is that the tradeoff to get off-peak pricing is that on-peak is way, way more expensive. So you can charge the EV off-peak to maximize the savings, but everything else you do during on-peak time costs way more.

Using myself as an example:

I adjust my A/C to run outside of 5pm-9pm (peak) if at all possible, we try to avoid pointless high-draw usage during that same window, and both of our EVs hold off charging until after 9pm.

My rate from 5pm-9pm is 0.43/kWh. My rate after 9pm is 0.09/kWh. The flat rate alternative, if I did not want to worry about time of day, would be 0.21/kWh. These prices are all-in, including transmission and distribution/whatever.

It would be dishonest to say that my EVs only cost me 0.09/kWh to operate, which on it's face is a claim to paying over 50% less. In reality, time of day pricing typically saves me somewhere between 10% and 15% in an average month compared with flat rate.

[mindslight]

If you leave the EV charging out of your consumption, does a time of day plan still save you money on the remaining usage? Or does it cost you? If it saves you money, then it would make sense to be on a ToD plan regardless of EV charging. Which means it makes sense to consider your additional EV draw as costing the marginal off-peak rate. Essentially the EV load has the valuable property of being dispatchable.

You can do the same thought experiment with say a dehumidifier in your basement. It can easily be off during peak usage and still accomplish its job, so its cost of electricity is also the marginal off-peak rate.

[searealist]

Fair point.

I guess it depends on if you would be using ToU otherwise.

It looks like about 50% of Californians use ToU plans, but the number is only 10% nation-wide.

[nightski]

Not the parent, but here is one location that has rates in that range in the U.S.

https://casscountyelectric.com/rates

[flutas]

I'm in Arkansas and get rates fairly similar as quoted.

From my last bill

> KWH USAGE 2590 - $183.37

There's a base customer cost of $18 on top of that, but yeah ~$0.077/kWh taxes included.

[mambru]

Is that for a month or a year?

[seafoam]

My monthly bills are similar to this, near Arkansas.

Last one was $212 for 2,146 kwh between June 8 - July 6 (28 Days)

[hdgvhicv]

76kWh a day, does that include a lot of EV mileage?!

[bloggie]

I pay a little less than that though I’m limited to 50 kW average load. Delivery is a fixed monthly fee, about $10 USD/month.

[tda]

To me (EU) that seems like a pretty generous limit. A normal house connection here is limited to 8kW single phase or 17kW for a 3 phase connection. You can get more, but that is very uncommon and gets expensive quickly

[nextaccountic]

> That's an unusually low electric rate for the US

What about buying your own solar panels and batteries?

[bravoetch]

Fort Collins, CO is probably where that rate is. I don't know of anywhere cheaper for residential electric.

[peterleiser]

Nashville Electric Service rates are 12.0 cents / kWh. Flat rate, no peak or off peak rates for residential.

[AgentMatt]

They are most likely not based in the US, but converting to USD to make comparison easier.

[walrus01]

I am not in Quebec but Quebec hydro rate D for standard residential would be one example of around what I pay.

https://www.hydroquebec.com/residential/customer-space/rates...

Another example would be Manitoba hydro

All figures in Canadian currency

https://www.hydro.mb.ca/account/rates/residential/

[brailsafe]

Specifying USD is indeed often a service usually offered by people born elsewhere for people born elsewhere. Americans seem rarely know about these mysterious places, where bills can come in all sorts of funny sizes and colours. (kind of joking)

[light_hue_1]

As someone who has worked in two industries that are at the maximal end of data sensitivity and privacy this comes across as a tinfoil hat issue not a real business requirement. In such cases we've always found ways to trade dollars for the privacy we need without having to run our own inference at excruciating slow speeds.

[walrus01]

Do you mean by trading dollars for the privacy you need as:

a) Contracting with a third-party independent inference provider who will run your choice of model on fast hardware that they own, with all appropriate data security/privacy/contractual/compliance protection in place

or

b) Contracting with the original creators of the model to run inference via their API and with assurances that all the same data protection is in place

or

c) Spending the money to buy your own inference hardware to run it on something you fully own/control at proper usable speeds?

Edit: Everything I've been writing in this thread is mostly within the context of being able to evaluate K3 and its usefulness to be self-hosted as a preliminary proof of concept or test of feasibility of a new thing, such as on <$20,000 of server hardware, before proceeding to spend 300-400k on GPU-related hardware, or external third party services/ongoing billing.

[jmalicki]

A) is very doable with e.g. Amazon Bedrock.

They'll give you HIPAA compliance, they even have a data center for US government classified data, they can give you European data sovereignty. And with OpenAI and Anthropic models to boot, you don't even have to settle for open weights.

What kind of privacy needs do you really have beyond that?

[walrus01]

It is not my use case but given recent political developments in international relations caused by the executive branch of the US government, off the top of my head, I could think of a lot of European or Canadian firms for which that would not be an option. No matter what they might promise about European sovereignty. For a good 'ol patriotic US domestic company? Sure.

[Taunt4]

Yes, its the US cloud act risk EU companies run up against on hyperscalers like MS/AWS.

Even for EU companies running open weights on EU stacks LLM inference on the GPU must process plaintext and I can't find any EU provider with NVIDIA H100/H200/Blackwell CC mode plus SEV-SNP or TDX, where you can cryptographically verify the workload ran somewhere the operator cannot inspect.

Personal compute is therefore the only option if you want personal autonomy privacy for IP &c. Maybe another option is to use cloud compute rented to fine tune a personal model that suits your own needs that would help bring the cost down, I don't know enough about this area to know if it kills the "intelligence" of those domains due to limited ?cross-verification within the LLM.

[SoftTalker]

It's also worth considering what you are actually paying for. And it's not keeping the data private, it's taking the blame when there is a breach. Same reason companies hire big consulting firms whenever they need to make an important but possibly risky decision.

[well_ackshually]

>very doable with <US company>

for anyone not US-based, this company is hostile and you have to assume the US government can and will force them to give access to your data.

[solarengineer]

There are regulated sectors in countries where data sovereignty is important enough that the sector sticks to air-gapped on-prem hardware and does not use cloud services at all. They have the dollars to pay for more than what it would cost to run on the Cloud.

[frognumber]

Having worked in / adjacent several such industries, a lot of the question depends on scale.

A trillion-dollar business can easily trade dollars for the privacy. A business with $1M to spend won't even get a phone call with OpenAI or Anthropic, who were the only* previous players in town for doing this.

Worst-case example: Bootstrapped startup working in military.

It's also the case that an open model enables many more intermediate-cost solutions. E.g. providers certified for specific applications, on-prem rentals, etc.

* Omitting Azure, which gives some privacy for some $$$ on their models, but not at the level of high-security.

[jmalicki]

> Worst-case example: Bootstrapped startup working in military.

That's the easiest case.

AWS Bedrock models running in AWS Secret Cloud for Industry. (I really have no affiliation with them, I'm just like... this is a completely solved problem, why do people think this is hard and requires on-prem hardware?)

https://www.aboutamazon.com/news/aws/aws-secret-cloud-for-in...

I'm with GP that these are tinfoil hat concerns, when there are solutions to all of these, unless you're perhaps in some country with very specific needs beyond things like European sovereignty or US military secrets (like a non-US defense concern).

[walrus01]

You seem to be categorizing everything that considers their data being in the possession of the US an unacceptable risk to be tinfoil hat, which is kind of an insult to a large portion of the world. If you haven't been paying attention to the news in the last 48 months, the political reality has shifted considerably.

Note that the other commenter never said US-based military oriented startup. You just assumed, then jumped to "heck yeah let's use Amazon Secret Cloud for Industry"

Not everyone has or wants an office in Crystal City.

[Sanzig]

I am in Canada. I cannot legally host defence workloads in the United States, even if I wanted to (I most certainly don't).

[amluto]

> Omitting Azure, which gives some privacy for some $$$ on their models, but not at the level of high-security.

If I were ranking third parties on their ability to safely handle my data without compromising it, I would rank Anthropic pretty low for things like Fable (where they more or less promise that they will misuse my data), but I want Azure pretty low in the sense that I fully expect them to be compromised.

I would tend to trust Amazon to avoid being compromised.

[frognumber]

At least for regulated applications I worked on, no one cared.

The provider needs to comply with specific rules, have specific certifications, and sign specific agreements. You check the boxes, and you're good to go.

Microsoft does that better than anyone. OpenAI and Anthropic don't do that at all. Google does that rarely and poorly. AWS is not bad, but not as good as Microsoft.

Azure was always my go-to for regulated applications in the cloud. Some do require e.g. on-prem or even air gap, where even Azure is out.

[fc417fc802]

The expectation that one BigTech company has a competent security team while the other doesn't seems entirely baseless?

[amluto]

https://windowsforum.com/security-alerts.84/cve-2025-55241-e...

https://orca.security/resources/blog/autowarp-microsoft-azur...

https://petri.com/microsoft-azure-automation-cross-tenant-ac...

I don’t recall much in the way of AWS vulnerabilities like these, but they’re a regular occurrence on Azure.

[ClumsyPilot]

There are situations where you don’t care about government spying but care about random criminals And there are situations that are the opposite

[trollbridge]

Interesting. So nobody would have had a problem with you running stuff on Chinese AI providers?

I have some inference I simply don't want to run on OAI, Anthropic, or Google because I don't want to run afoul of their "rules" and end up with a banned account, and this situation is only getting worse when it comes to doing fairly basic tasks like trying to secure your app against security problems.

[fooker]

Great, so the other member of the set matters for you more than cost.

Do you actually need to run the state of art model at 5 tokens per second instead of a qwen or whatever 7b or 30b model at 100 tokens per second?

[walrus01]

Do I really need to? No, not really. The 27B full density, 35B MoE, 70B and 122B models I have in use get me 95% of the way there on a lot of things. Particularly when dealing with languages and systems where I have at least an intermediate level of knowledge on, to know whether something is going down a dead end, using a wrong method, metaphorically chasing its tail, or is producing valid output.

On the other hand, would it be cool to also have a really big thing as an ancillary tool that I could throw a request into opencode before going to bed, let it crank away and take a look at what it's done 7 hours later? Yeah, particularly if I (very much an unknown quantity at this time) could be confident that it builds high quality, syntax valid, appropriately commented and not absurd code.

[sm-silversight]

>Do you actually need to run the state of art model at 5 tokens per second instead of a qwen or whatever 7b or 30b model at 100 tokens per second?

Some people like doing things they want to do. Do I actually need to buy expensive pigments from europe to make paintings of flowers? My camera produces a much more accurate representation.

[walrus01]

Very good description of it. It does seem like a bit of a rhetorical question to ask a forum that has a very high population of Linux and BSD users why they might desire to have the option to do something themselves rather than relying on an external packaged ready to go product.

[irishcoffee]

The whole mentality of thinking one knows better than another about what they need causes infinitely more problems than it solves.

[ljlolel]

can send safely context if there’s confidential computing ala my site https://trustedrouter.com/

[skinfaxi]

How do you prove you are running exclusively on Nitro enclave instances or GCP confidential spaces?

[whs]

Seems clear from their website?

1. Their API server provide an attestation JWT. This JWT is signed by Google's private key. 2. The attestation has details on the running container. I suppose the container host is a Google-provided distro and Google's signer will verify that the OS is theirs and up-to-date. 3. They could've proxy the attestation. To prove this is not the case, the field eat_nonce include the TLS certificate fingerprint, which should match the API server you're connecting to. I suppose you will need to pull their container and verify from the source that the container itself generate the private key, it never leaves the container, and the container has no way to run arbitrary code such as SSH or vulnerabilities.

[Cider9986]

Is your local compute airgapped?

[trollbridge]

My local compute is used by me, and I'm accountable to myself whether or not is secure. So to a certain degree, I trust myself and also know what limitations / potential vulnerabilities it might have.

[well_ackshually]

I trust my local compute quite a bit more than a random project that has "trusted" in its name.

[ComputerGuru]

> I pay about $0.075 USD per kWh

Around here electricity companies quote prices like yours but that is supply only while transmission, taxes, and fees are again as much on top. Is that really all inclusive?

[pizza234]

Which are those use cases, considering that the hypothetical server, as described by the GP, is also extremely slow (4/6 hours for a response)?

[doctorpangloss]

It's completely academic. At 5tok/s you can process 13 MTok per month at concurrency 1. I use 5 BILLION tokens per week when coding.

[trollbridge]

Yeah. At 5 tok/second, you're talking about around $195 worth of output tokens per month. There is no way I can run a usable K3 model for $195 a month of capex, opex, or any-kind-of-ex.

Qwen 3.6 is another matter. Paying provider rates for the amount I run locally would put me in the thousands of dollars. So that's very practical to buy a Macbook instead, plus an RTX card, and so on.

[polycancel]

>and people will compromise speed for data sovereignty

People should always compromise speed for data sovereignty! Who said: that in this digital day and age, information about money is more important than money!

[tiahura]

There are a number of use cases where sending the contents of your context and prompts (and the resulting output) to a 3rd party service is off the table as an option, and people will compromise speed for data sovereignty.

Are there? At the highest levels of defense and law, AWS and Azure are used.

Having tried selling some of these entities on doing things in-house, there seems to be little interest.

[walrus01]

> Are there? At the highest levels of defense and law, AWS and Azure are used.

This is certainly true if the user is an American company. You could look at the European initiatives to run this stuff on hardware they own in facilities they own and control within the borders of Europe for a counter-example.

Such as: https://www.google.com/search?client=firefox-b-d&q=schwarz+s...

https://www.dutchnews.nl/2026/04/government-turns-to-german-...

[jmalicki]

Yeah, true European cloud providers for these kinds of things seem to be behind, and a lot of the ones offering data compliance at the level of AWS are small enough that it's a bit harder to trust they'll be around and will keep their promises.

Hopefully that changes!

[cobbzilla]

I’ve priced it out: max $135/month to run a dual Xeon 2U server with 3T RAM & 2x 22 core Xeon Gold. It’s the 2x 750W power supplies that ultimately determine opex. My power costs $0.124/kWh, the $135 assumes drawing maximum power continuously, and in that case, I can probably offset my heating bill a little bit in the winter, so maybe effectively a little bit lower.

I don’t know if that’s 100x more than I’d pay (opex-wise) with an nvidia setup, but I can say the one-time capex is a great deal cheaper. Avoiding VRAM and DDR5 (fast DDR4 should be OK) are the biggest cost savers. ECC RAM is worth the extra price. General datacenter-quality hardware has less price sensitivity, and plenty of bang for your buck.

[walrus01]

Keep in mind that just because it has dual 750W power supplies that doesn't mean it's what its load will be, for a full CPU loaded wattage figure you'd need basically a pair of kill-a-watts plugged in inline on the feed for each poewr supply and then run stress-ng with artificial cpu stress on all cores for an hour.

Under heavy inference load you will find that the cpu usage is actually less as the bottleneck is the RAM bus speed. An older 2U rack server that is 600W load (typically a 1+1 power supply server when plugged into two kill-a-watt would show 300W on each, equal load balancing) when maxed out with stress-ng might be only 450W total running inference.

If you have 600kWh used in a month by running something 24x7 and your power is $0.15 a kWh, that's more like $90/mo (not counting cooling or any ancillary costs for the environment where it's in).

[trollbridge]

If you actually were running this thing at 80% or 100% load, then the first thing you'd want to is get a better PDU and then connect your servers to that (48V DC).

[Kirby64]

Why? Standard server grade PSUs are 80 Plus Platinum, exceeding 95% efficiency. The whole server is designed around 12V, especially for older gear, and replacing it with components that are 48V would be wildly expensive relative to the efficiency benefits.

[walrus01]

One of the problems in buying used/refurb x86-64 rack servers for test and development/proof of concept environment, is that by volume in the market, there's not that many -48VDC power supplies going around, because maybe 5-10% of enterprise customers buy them. Resulting in many fewer units ending up on the resale market.

The options for AC power supplies for servers with 2 or 4 load sharing redundant power supplies are a lot greater. If you were buying all new hardware and starting from a clean sheet of paper design with lots of money to spend, absolutely. At that point also start looking at higher voltage DC distribution stuff related to open compute platform and 800VDC.

But if I were trying to make the absolute most use of $20,000 to put together a 3TB RAM server (48 x 64GB DIMMs), it would likely end up AC powered.

[sebmellen]

(Context: Parent comment was edited after I wrote this comment)

Where in the world are you finding that much RAM in a racked server for $200/month?

[walrus01]

I think he means electrical bill at his estimated wattage load of the server and his known kWh cost, not rented server/hosting cost.

[sebmellen]

Aha, right. That makes a lot more sense.

[cobbzilla]

At my house. I have 5Gbps fiber and could pay for 10 or 25 if I need it.

[sebmellen]

Gotcha. But to be clear, you’re talking only about energy usage, correct?

[cobbzilla]

Yes, what other opex is there? It will have good ventilation, I’m not worried about cooling.

[sebmellen]

I thought you were talking about renting a dedicated server from a hosting provider, not racking it yourself. My bad!

[btown]

One aspect of this is cyberattack proliferation by way of "Hey boss, I saw this TikTok that says if you let me invest [a tiny piece of the neighborhood's profit|our militia's budget] into some RAM, I could get a fully autonomous cyber operation up and running that pays for itself via ransomware etc. within weeks. You like it, we upgrade to something that can work even faster. We don't need the hacker guy from Swordfish with fifty monitors, we just need my cousin who likes building gaming PCs."

That's a world that I don't think we're ready for.

[9x39]

A similar world is already here.

Young men 14-?? already compromise and attempt to extort organizations daily, sometimes cluelessly from western nations, often not. It doesn’t have to be gangs when the home country doesn’t care / isn’t technologically or culturally developed.

Already seeing AI-written payloads and frameworks in the wild. I think it’ll turn out that AI won’t build you a maintainable ERP but it can create C2 networks, exploit POCs or even 0-days potentially, and let kids make their own ransomware tooling. Then we’re dealing not with a handful of cybercrime tool makers but a generational problem.

[tripzilch]

> that justifies this

this is hacker news sir

we don't do things because they are easy ... or because they are justified ...

[tripzilch]

> extreme data privacy requirement

so I have a a few decades worth of creative coding projects, many personal notes + writing, pictures, video clips, renders, projects, backups of the previous laptop's home folder that contains the laptop before it in the ~/dump folder and so on, spread out over several harddisks, laptops and storage media

it's a mess that I've wanted to organize for years, but I can't seem to manage it

but I might be able to sketch out a plan or task for a local multi-modal LLM to slowly chug through all the files (read-only), catalog and index them, copy to a clean external SSD, de-dupe and tidy everything up

I mean, some of this data is already highly personal, and there might be stuff I forgot was there

... is it an "extreme data privacy requirement" to not want to share running an LLM for this task, and wanna do it locally?