[bryan0]

> Yeah, this is anthropic advocating for a ban on open weight models.

This is an ungenerous take, and I think it's important to to recognize it's reasonable to support models that are both open and safe. How this would actually be achieved is unclear though. Dario is at least proposing a solution a solution, which is the model needs to pass safety testing. This is reasonable and I wouldn't conflate this with wanting to ban open weights.

I think the deeper problem might be though that once you have safe open-weight models, it will be much easier to make them unsafe. And to be specific, unsafe means proliferation of chemical, biological, radiological, and nuclear (CBRN) weapons knowledge and similar information.

[jjfoooo4]

> How this would actually be achieved is unclear though. Dario is at least proposing a solution a solution

How it would be achieved is a pretty important bit! One which Dario is not proposing any concrete solution for other thanks hand waves at some gov safety committee.

Would this restrict downloads of an open model, or publishing?

Say we ban domestic hosting un-approved open models. How does Dario propose to ban downloads from abroad? You can’t tell what an encrypted payload contains, do we need to restrict encryption?

[jsnell]

Isn't it up to the open model advocates and publishers to come up with the solutions for making them safe?

Like, there's three plausible arguments about safety of open models:

1. Any concerns are fake news. Open models will always be safe.

2. Safety is irrelevant. Open models should not be regulated even if they're unsafe.

3. Safety is a technical problem with technical solutions. People releasing open models should invent and implement such solutions.

I think option 1 is totally out of touch with reality.

Option 2 is at least self-consistent, it's the argument being made by people who will say that all regulation is always bad. It's also like the worst possible world from an x-risk perspective (but I realize that the average HN poster believes any x-risk concerns are just frontier lab marketing).

Option 3 is playing on hard mode compared to proprietary models, which can both implement additional safeguards out-of-model and prevent modifications of the model. But if the answer to it is "it's too hard, Anthropic needs to come up with the technical solution", then that's not exactly a ringing endorsement for the safety practices of the open model labs, right?

[cogman10]

In order for model safety regulation to be effective, you need everyone capable of producing models to sign on to that safety framework.

That will never happen.

As such, there is no "solution" here.

The best most perfect regulation in the US won't prevent a malicious actor in the US from running a dangerous model. It's simply too easy to VPN to a country that doesn't care about AI safety and to run or download that model and run it in the US.

There's no solution to this, which is why option 2 is the only option. The only thing safety regulations can possibly do is blunt the usage of "unsafe" models. And the primary people that will be blunted by it are people that do not and would not use these unsafe models in an unsafe fashion.

It's not that I think regulation is always bad/wrong whatever, I'm no libertarian. But I also recognize when regulation is pointless. You can't regulate away forbidden knowledge, which is effectively what a dangerous model is.

[cogman10]

> This is an ungenerous take

Why should I give a multi-billion dollar company advocating for new regulations in its industry a generous take?

I'd be similarly cynical if McDonald's proposed new health and safety regulations for restaurants.

[cloverich]

There is an important difference between criticism and cynicism.

[0xDEAFBEAD]

>I'd be similarly cynical if McDonald's proposed new health and safety regulations for restaurants.

"Because McDonald's wants food regulations, we can therefore conclude that all food regulations should be eliminated."

Obviously this would be rather silly.

It would be helpful to stop obsessing about McDonald's finances and simply discuss the best food regulation strategy. We just can't learn all that much about the best way to regulate food by making cynical proclamations about which food regulations will benefit the bottom line at McDonald's.

[cogman10]

> Obviously this would be rather silly.

Which is why it wasn't the point I was making. You did an uncharitable reading of my position and then did a straw man attack.

My position is that any food regulation the likes of McDonald proposes should be looked at in the most critical and cynical light possible. They aren't making such proposals for the general health of the public, but rather to improve their own bottom line.

My position is not and never was that "we should not regulate food".

> It would be helpful to stop obsessing about McDonald's finances and simply discuss the best food regulation strategy.

McDonald's uses their market position and wealth to directly lobby to government officials about food regulations. I worry about what McDonald's has to say about food because they have a VASTLY outsided ability to manipulate the regulatory system.

> We just can't learn all that much about the best way to regulate food by making cynical proclamations about which food regulations will benefit the bottom line at McDonald's.

We can call out ineffectual and blatently self serving calls for new regulations for what they are, McDonald's trying to use regulatory capture to increase their profits and hurt their competitors.

Back on topic, that's exactly the situation with open ai.

IMO, this isn't something that's regulatable because AI models are ephemeral data that's easy to copy and replicate. No amount of US regulations can stop China from sending their dangerous models to Iran. The only thing such draconian measures accomplishes is building a moat for the likes of anthropic to shrink the number of potential customers.

If we must push out laws around AI, then those laws should at least have some chance of success. I'm all in favor of criminalizing the use of AI in cyber attacks, scamming, etc. But that's a capability that is model agnostic.

Much like I'm in favor of health and safety checks on a restaurant but I think having a mandatory McDonald's built and sold food safety device in every restaurant would be nuts. It wouldn't make food healthier it'd only serve to benefit McDonald's bottom line.

[parineum]

> This is an ungenerous take

I think that's well earned.