[hyeonwho22]

Why not? It has already been shown that AI can be (mis)used to identify good candidates for chemical weapons. [1] Next in the pipeline is obviously some religious nut (who would not otherwise have the capability) using it to design a virus which doesn't set off alarms at the gene synthesis / custom construct companies, and then learning to transfect it.

More banally, state actors can already use open source models to efficiently create misinformation. It took what, 60,000 votes to swing the US election in 2016? Imagine what astroturfing can be done with 100x the labor thanks to LLMs.

[1] dx.doi.org/10.1038/s42256-022-00465-9

[visarga]

> Next in the pipeline is obviously some religious nut (who would not otherwise have the capability)

So you're saying that:

1. the religious nut would not find the same information on Google or in books

2. if someone is motivated enough to commit such an act, the ease of use of AI vs. web search would make a difference

Has anyone checked how many biology students can prepare dangerous substances with just what they learned in school?

Have we removed the sites disseminating dangerous information off the internet first? What is to stop someone from training a model on such data anytime they want?

[hyeonwho22]

1. The religious nut doesn't have the knowledge or the skill sets right now, but AI might enable them.

2. Accessibility of information makes a huge difference. Prior to 2020 people rarely stole Kias or catalytic converters. When knowledge of how to do this (and for catalytic converters, knowledge of their resale value) became available (i.e. trending on Tiktok), then thefts became frequent. The only barrier which disappeared from 2019 to 2021 was that the information became very easily accessible.

Your last two questions are not counterarguments, since AIs are already outperforming the median biology student, and obviously removing sites from the internet is not feasible. Easier to stop foundation model development than to censor the internet.

> What is to stop someone from training a model on such data anytime they want?

Present proposals are to limit GPU access and compute for training runs. Data centers are kind of like nuclear enrichment facilities in that they are hard to hide, require large numbers of dual-use components that are possible to regulate (centrifuges vs. GPUs), and they have large power requirements which make them show up on aerial imaging.

[api]

What happens if someone develops a highly effective distributed training algorithm permitting a bunch of people with gaming PCs and fast broadband to train foundation models in a manner akin to Folding@Home?

If that happened open efforts could marshal tens or hundreds of thousands of GPUs.

Right now the barrier is that training requires too much synchronization bandwidth between compute nodes, but I’m not aware of any hard mathematical reason there couldn’t be an algorithm that does not have to sync so much. Even if it were less efficient this could be overcome by the sheer number of nodes you could marshal.

[hyeonwho22]

Is that a serious argument against an AI pause? There are potential scenarios in which regulating AI is challenging, so it isn't worth doing? Why don't we stop regulating nuclear material while we're at it?

In my mind the existential risks make regulation of large training runs worth it. Should distributed training runs become an issue we can figure out a way to inspect them, too.

To respond to the specific htpothetical, if that scenario happens it will presumably be by either a botnet, by a large group of wealthy hobbyists, or by a corporation or a nation state intent on circumventing the pause. Botnets have been dismantled before, and large groups of wealthy hoobyists tend to interested in self preservation (at least more so than individuals). Corporate and state actors defecting on international treaties can be penalized via standard mechanisms.

[api]

You are talking about some pretty heavy handed authoritarian stuff to ban math on the basis of hypothetical risks. The nuclear analogy isn’t applicable because we all know that a-bombs really work. There is no proof of any kind of outsized risk from real world AI beyond other types of computing that can be used for negative purposes like encryption or cryptocurrency.

Here’s a legit question: you say pause. Pause until what? What is the go condition? You can never prove an unbounded negative like “AI will never ever become dangerous” so I would think there is no go condition anyone could agree on.

… which means people eventually just ignore the pause when they get tired of it and the hysteria dies out. Why bother then?

[dingnuts]

>It took what, 60,000 votes to swing the US election in 2016?

this is a conspiracy theory that gained popularity around the election but in the first impeachment hearings, those making allegations regarding foreign interference failed to produce any evidence whatsoever of a real targeted campaign in collusion with the Russian government.

[CamperBob2]

Eh, he's not wrong there -- our elections are subject to chaos theory at this point, more than the rule of law. Sensitive dependence on both initial conditions and invisible thresholds. The Russian "butterfly effect" in 2016 was very real. Even if it wasn't enough on its own to tip the balance in Trump's favor, they were very clear that Trump was their candidate of choice, and they very clearly took action in accordance with that. Neither of these statements is up for the slightest debate at this point.

However, the possibility of foreign election interference, real or imagined, is not a valid reason to hold back progress on AI.

[tsurba]

Essentially you are advocating against information being more efficiently available. Come on.

It’s true we are fucked if bioweapons become easy to make, but that is not a question of ”AI”.

[hyeonwho22]

The only thing keeping bioweapons from being easy to make is information becoming easily available.

> Essentially you are advocating against information being more efficiently available.

Yes. Some kinds of information should be kept obscure, even if it is theoretically possible for an intelligent individual with access to the world's scientific literature to rediscover them. The really obvious case for this is in regards to the proliferation of WMDs.

For nuclear weapons information is not the barrier to manufacture: we can regulate and track uranium, and enrichment is thought to require industrial scale processes. But the precursors for biological weapons are unregulated and widely available, so we need to gatekeep the relevant skills and knowledge.

I'm sure you will agree with me that if access information on how to make a WMD becomes even a few order of magnitudes as accessible as information on how to steal a Kia or how to steal a catalytic converter, then we will have lost.

My argument is that a truly intelligent AI without safeguards or ethics would make bioweapons accessible to the public, and we would be fucked.