[light_hue_1]

As someone who has worked in two industries that are at the maximal end of data sensitivity and privacy this comes across as a tinfoil hat issue not a real business requirement. In such cases we've always found ways to trade dollars for the privacy we need without having to run our own inference at excruciating slow speeds.

[walrus01]

Do you mean by trading dollars for the privacy you need as:

a) Contracting with a third-party independent inference provider who will run your choice of model on fast hardware that they own, with all appropriate data security/privacy/contractual/compliance protection in place

or

b) Contracting with the original creators of the model to run inference via their API and with assurances that all the same data protection is in place

or

c) Spending the money to buy your own inference hardware to run it on something you fully own/control at proper usable speeds?

Edit: Everything I've been writing in this thread is mostly within the context of being able to evaluate K3 and its usefulness to be self-hosted as a preliminary proof of concept or test of feasibility of a new thing, such as on <$20,000 of server hardware, before proceeding to spend 300-400k on GPU-related hardware, or external third party services/ongoing billing.

[jmalicki]

A) is very doable with e.g. Amazon Bedrock.

They'll give you HIPAA compliance, they even have a data center for US government classified data, they can give you European data sovereignty. And with OpenAI and Anthropic models to boot, you don't even have to settle for open weights.

What kind of privacy needs do you really have beyond that?

[walrus01]

It is not my use case but given recent political developments in international relations caused by the executive branch of the US government, off the top of my head, I could think of a lot of European or Canadian firms for which that would not be an option. No matter what they might promise about European sovereignty. For a good 'ol patriotic US domestic company? Sure.

[Taunt4]

Yes, its the US cloud act risk EU companies run up against on hyperscalers like MS/AWS.

Even for EU companies running open weights on EU stacks LLM inference on the GPU must process plaintext and I can't find any EU provider with NVIDIA H100/H200/Blackwell CC mode plus SEV-SNP or TDX, where you can cryptographically verify the workload ran somewhere the operator cannot inspect.

Personal compute is therefore the only option if you want personal autonomy privacy for IP &c. Maybe another option is to use cloud compute rented to fine tune a personal model that suits your own needs that would help bring the cost down, I don't know enough about this area to know if it kills the "intelligence" of those domains due to limited ?cross-verification within the LLM.

[SoftTalker]

It's also worth considering what you are actually paying for. And it's not keeping the data private, it's taking the blame when there is a breach. Same reason companies hire big consulting firms whenever they need to make an important but possibly risky decision.

[well_ackshually]

>very doable with <US company>

for anyone not US-based, this company is hostile and you have to assume the US government can and will force them to give access to your data.

[solarengineer]

There are regulated sectors in countries where data sovereignty is important enough that the sector sticks to air-gapped on-prem hardware and does not use cloud services at all. They have the dollars to pay for more than what it would cost to run on the Cloud.

[frognumber]

Having worked in / adjacent several such industries, a lot of the question depends on scale.

A trillion-dollar business can easily trade dollars for the privacy. A business with $1M to spend won't even get a phone call with OpenAI or Anthropic, who were the only* previous players in town for doing this.

Worst-case example: Bootstrapped startup working in military.

It's also the case that an open model enables many more intermediate-cost solutions. E.g. providers certified for specific applications, on-prem rentals, etc.

* Omitting Azure, which gives some privacy for some $$$ on their models, but not at the level of high-security.

[jmalicki]

> Worst-case example: Bootstrapped startup working in military.

That's the easiest case.

AWS Bedrock models running in AWS Secret Cloud for Industry. (I really have no affiliation with them, I'm just like... this is a completely solved problem, why do people think this is hard and requires on-prem hardware?)

https://www.aboutamazon.com/news/aws/aws-secret-cloud-for-in...

I'm with GP that these are tinfoil hat concerns, when there are solutions to all of these, unless you're perhaps in some country with very specific needs beyond things like European sovereignty or US military secrets (like a non-US defense concern).

[walrus01]

You seem to be categorizing everything that considers their data being in the possession of the US an unacceptable risk to be tinfoil hat, which is kind of an insult to a large portion of the world. If you haven't been paying attention to the news in the last 48 months, the political reality has shifted considerably.

Note that the other commenter never said US-based military oriented startup. You just assumed, then jumped to "heck yeah let's use Amazon Secret Cloud for Industry"

Not everyone has or wants an office in Crystal City.

[Sanzig]

I am in Canada. I cannot legally host defence workloads in the United States, even if I wanted to (I most certainly don't).

[amluto]

> Omitting Azure, which gives some privacy for some $$$ on their models, but not at the level of high-security.

If I were ranking third parties on their ability to safely handle my data without compromising it, I would rank Anthropic pretty low for things like Fable (where they more or less promise that they will misuse my data), but I want Azure pretty low in the sense that I fully expect them to be compromised.

I would tend to trust Amazon to avoid being compromised.

[frognumber]

At least for regulated applications I worked on, no one cared.

The provider needs to comply with specific rules, have specific certifications, and sign specific agreements. You check the boxes, and you're good to go.

Microsoft does that better than anyone. OpenAI and Anthropic don't do that at all. Google does that rarely and poorly. AWS is not bad, but not as good as Microsoft.

Azure was always my go-to for regulated applications in the cloud. Some do require e.g. on-prem or even air gap, where even Azure is out.

[fc417fc802]

The expectation that one BigTech company has a competent security team while the other doesn't seems entirely baseless?

[amluto]

https://windowsforum.com/security-alerts.84/cve-2025-55241-e...

https://orca.security/resources/blog/autowarp-microsoft-azur...

https://petri.com/microsoft-azure-automation-cross-tenant-ac...

I don’t recall much in the way of AWS vulnerabilities like these, but they’re a regular occurrence on Azure.

[ClumsyPilot]

There are situations where you don’t care about government spying but care about random criminals And there are situations that are the opposite

[trollbridge]

Interesting. So nobody would have had a problem with you running stuff on Chinese AI providers?

I have some inference I simply don't want to run on OAI, Anthropic, or Google because I don't want to run afoul of their "rules" and end up with a banned account, and this situation is only getting worse when it comes to doing fairly basic tasks like trying to secure your app against security problems.