[chrsw]

Correct. We need open weights, open code and open data. If nobody else can reproduce what someone did there will always be security questions. Even if we can reproduce it there could still be security concerns but it's more realistic to investigate yourself.

[nl]

I'm all for open models, but people seem to misunderstand what they are. They aren't the same thing as open source code!

> open weights, open code and open data

Even if you have all these things you still can't replicate a model because of randomness.

You can backdoor a model with less than 1000 examples and it is impossible to detect.

[2sk21]

Yeah - we also don't know if the models from OpenAI and anthropic are back-doored either.

[chrsw]

You don't want to replicate the exact model, you want to build a system of similar capabilities.

[nl]

Great, but that seems a different concern to the auditability of a model.

You can take the code for Kimi K3 now, take the training framework from Prime and the data from Olmo, spend some money on RL environments and some more money (!) on GPU training and end up with a system of similar capabilities.

But that's completely different to being able to audit Kimi K3. Even if you had the exact code, data and training environments it is impossible to verify that the model you have came from that.

[Ericson2314]

Deterministic seed

[nl]

Deterministic seeds barely work on a single machine, small scale training run.

They just don't work at all on a many month long, 100K+ GPU cluster training run.

[fc417fc802]

While using floating point? Not happening. You'd have to switch to fixed point, not just for the models themselves but also _all_ the training code (ie backprop).

Even then you'd still need to account for order of events when an entire cluster of GPUs is involved. Also don't forget to account for any synthetic data sources. Or even non-synthetic for that matter - does your pipeline do any image resizing on the fly? Better make sure that's fully deterministic between machines (it almost certainly won't be).

It's theoretically possible but I don't expect it to materialize any time soon.

[nl]

> theoretically possible

I mean I guess, but not in a performant way if there are ever any hardware failures. And with 100K GPUs there are multiple hardware failures per day.

[essentia0]

Exactly what are the possible 'security issues' of self hosting an open weights model?

[perching_aix]

It may have been backdoored during training, potentially causing it to randomly start wreaking havoc at runtime, possibly in a clandestine manner (e.g. sneaking in bugs into generated code).

[miyoji]

This isn't a security issue related to self-hosting, it's a security issue related to use and it is shared entirely by closed weights models. Anthropic could easily be sneaking bugs into your generated code, too.

[perching_aix]

Correct, that was not my point either.

[miyoji]

You were answering the question "What security issues come from self-hosting?" The security issue you named has nothing to do with self-hosting. What, then, was your point?

[perching_aix]

We seem to be reading the same comment(s) differently.

The context (verbatim):

> Correct. We need open weights, open code and open data. If nobody else can reproduce what someone did there will always be security questions. Even if we can reproduce it there could still be security concerns but it's more realistic to investigate yourself.

In short, it's an appeal to full openness and reproducibility on the basis of security; open weights alone notably do not provide that same confidence. They're better in some respects, not really in others.

Then comes the question (also verbatim):

> Exactly what are the possible 'security issues' of self hosting an open weights model?

Implying then that as long as you do have the weights and just self host it, the asker cannot imagine what could possibly go wrong. What is the gap, if any?

And so I explained. That was my point. Open weights do not give you full reproducibility, and so that on its own falls short of what the parent comment is making an appeal to. That there does remain a security concern, shared by remote and closed models, that does not improve just by having the weights, but would if you did have full reproducibility. Explaining that gap was my point, as that is what I understood as being asked there. It's the only thing I can reasonably imagine being asked, in fact.

This is a materially different question to what you apparently extracted (again, verbatim):

> What security issues come from self-hosting?

Implying that by self-hosting models, something bad might specifically happen.

I do not think this, do not think I suggested this, do not think the original question suggested this, and generally do not think this is indeed any sensible, in or outside the context. Certainly not beyond something common sense, like vLLM being compromised or whatever.

You seem to agree. But then how did we get here, clearly talking past each other?

[utilize1808]

e.g. be trained to favour including compromised dependences into your projects.

[urams]

> We need open weights, open code and open data.

Even with this, the cost of verification would be enormous. You would need a massive cluster to repeat the training E2E.