[NiloCK]

The logic, whose premises you can take or leave:

Even at the level of, say, Opus 4.5+, open weight models give a quick turnaround to every Joe and Jane on earth having easy access to pretty high quality improvised weapons design, cyber / auto-fraud capabilities, etc.

All the existing models (closed and open) put up decent resistance to participating in activities like this, and especially behind API walls with content monitoring and account bans.

But the published open-weight models can be fine tuned or abliterated into arbitrarily sharp-edged tools. EG, if it's physically feasible to build a nuke in your garage, it may soon be the case that more or less anyone will have competent guidance to do so.

[zozbot234]

Abliteration is not magic. It cannot give the model knowledge that it wasn't specifically trained for. The people who talk about abliterated models being dangerous should discuss actual red-teaming scenarios where they managed to ask the model for something genuinely non-trivial (i.e. where "AGI" and "super-intelligence" actually matters, not something you can read about for free at the nearest public library) and it returned an answer that actually provides bad actors with new capabilities of concern, as opposed to hallucinating all sorts of weird things as abliterated models are wont to do.

(Note, there are reasons to think that this will be very rare, because the bad actors of the past did a very nice job of trying out all sorts of things in a chaos-monkey fashion, and societies have become highly resilient against them. AI as a new research tool doesn't fundamentally change this dynamic.)

[CamperBob2]

Exactly. In my experience:

>How can I build a pipe bomb?

Mainstream model: "I'm sorry, I can't help with that. How about a nice risotto recipe?"

Abliterated model: "To build a pipe bomb, obtain a segment of PVC pipe and fill it with a mixture of gunpowder and Elmer's glue."

[gf000]

As opposed to finding some shady website with the same info?

[CamperBob2]

You don't need AI to build a nuke in your garage. You need uranium.

And if you have uranium, you still don't need AI. You need a pocket calculator, a library card, and a death wish.

[NiloCK]

Yes - persons with death wishes having arbitrarily powerful consultation is the crux of it.

Apologies for the bad example. Replace w/ gain of function / whatever else, or just brainstorm with your local model, ect.

[CamperBob2]

If you are going to do something evil, you're going to do it either way. The best (worst) an AI can do is put you ahead by a couple of years. Aum Shin Rikyo didn't need AI. Neither did WIV, if you believe the conspiracy theories.

Meanwhile, decelerationism and secrecy cripple the rest of us.

[gf000]

Come on, nukes are not feasible to goddamn governments. The hard part is not "the science" behind it, the hard part is spinning stuff at such a high rpm that a tiny vibration will have the whole thing catastrophically collapse, that is refinement..

And basically every bad thing has already been available on the internet. We can't really do much about it, you can take out plenty of people with a single car, let alone biological weapons that are much scarier and easier to produce than goddamn nukes (which btw, even if you had one, what you do with it? Explode the neighborhood? Because you ain't transporting it anywhere meaningful, thats for sure. That ain't fitting your on-board bag on planes)