[stratos123]

> Who should we fear more? All of collective humanity with the keys to build destructive (and defensive) stuff with AI, or small groups of elites, billionaires, and state actors who have the monopoly on violence and want to control the keys?

I'd trust the latter and I think it's an easy choice. I'm sure it feels bad to not be in the group with access to the scary weapons, but do remember that out of your two groups, the "collective humanity" one is the one with the literal terrorists, which do in fact exist and aren't a myth. For comparison, observe how the concentration of ability to produce nuclear weaponry in the hands of only a few states did, historically, work to prevent both a nuclear war and any nuclear terrorism.

> And the rest of us have no way to stop it or defend.

If you did have access, what'd be your defense plan? Biorisk is one of the most attacker-favoring fields imaginable, so a world where there's an equilibrium between attackers and defenders in bioweaponry research (the same way cybersecurity currently works) would be quite terrible. Your best bet would probably be to take a new vaccine each time a new engineered disease comes out, and hope that you're never one of the suckers who got infected before the vaccine was developed, and that the accumulated side effects from multiple experimental vaccines don't kill you too quickly.

[txrx0000]

I often see this parallel being drawn between general intelligence and nuclear weapons, and you seem genuinely concerned, so I'll try to explain why this concern is misplaced.

Intelligence is not a single-use weapon of mass destruction in the same way that nuclear weapons are. If I blow up a nuke and it vaporizes downtown, there's no "reverse-nuke" I can blow up afterwards to undo the explosion of the first nuke, or a "nuke nullification field" that I can set up beforehand to protect an area from the effects of a later explosion. But the equivalent of all of these things exist for general intelligence and the adversarial/cooperative dynamics that arise thereof such as bio/cyber attack/defense. We're just currently too stupid to realize how that would work, kinda like how monkeys mostly don't understand how humans resolve conflict, and that's why we see a temporary asymmetry in these strategies.

But it must be symmetric in the long run, because two things are simultaneously true: more intelligent species are able to better cooperate and prosper, and the intelligence difference is very small between individuals within the same species. This pattern is generally true from humans to animals to insects, all the way down to bacteria. There's no reason to believe this deep evolutionary pattern will suddenly become false. It has persisted through plenty of qualitative transitions in the past.

So the primary risk of the upcoming intelligence explosion is not adversarial dynamics such as bio/cyber risk. The primary risk is humanity splitting into two or perhaps more species, with the top superintelligent species preying on the rest (though it's also possible that we become cells in larger government/corporation-sized organisms). Broad diffusion of general intelligence is what can prevent this.

[elliotec]

It should not be an easy choice to trust the latter. Sure, "literal terrorists" exist, but guess what? State terrorism (and state-sponsored terrorism) in the form of massacres, genocide, torture, forced disappearances, etc. is far more deadly and far more prevalent than fringe ideological groups randomly attacking civilians.

These states and elites with the access to the scary weapons ARE the ones doing the damage. The call is coming from inside the house.

I don't understand your point on nuclear proliferation concentrated in the hands of a few states preventing nuclear war or terrorism. Remember, the only two nuclear weapons used in attacks killed a quarter million people, ~90-95% civilians. By one of the few states who had the power. This is the definitional paradigm of state terrorism. And like, now look at the status quo of nuclear treaties and agreements and proliferation. Not exactly a success story.

It's obviously not worth personally formulating a "defense plan" for an AI-enabled bio attack were I personally to have access to SOA weights, but if history is any indication, I feel pretty confident that the likelihood of that happening remains far greater in the closed-weight, elite-state-access-only scenario than the open, democratized, and collectivized one.

[stratos123]

> I don't understand your point on nuclear proliferation concentrated in the hands of a few states preventing nuclear war or terrorism. Remember, the only two nuclear weapons used in attacks killed a quarter million people, ~90-95% civilians.

Or, another way to put it: an extremely powerful military technology the careless usage of which could destroy the human civilization was only used in one war in history and killed less than a million people, and then it was never used again over the next 80 years. I think this is a success story for humanity, and the NPT was an amazing feat of coordination.

> And like, now look at the status quo of nuclear treaties and agreements and proliferation.

I mean, sure, but giving every state (much less every person) the ability to make nuclear weaponry wouldn't make things any better.

I think the reason we think of this differently is because you believe that states are by default... corrupt, maybe, or unreliable, or evil, while individuals are mostly fine. Whereas I think that most people can't be trusted to make correct decisions on topics like "should we use this dangerous technology", while states at least have some decent track record at this.

[elliotec]

Indeed, the core of this misalignment is that we have a foundational difference of opinion on that.

The only caveat I'd push is not that I believe that "individuals are mostly fine," but that as a whole, separate from the state apparatus, I believe humanity collectively has its own best interests in mind and is capable of achieving more good than states or coercive hierarchies can or want to achieve.

Thanks for the civil discourse!