[coryfklein]

If you think there is even a 5% chance that super-human level artificial general intelligence could be catastrophic for humanity, then the recipe for how to make AGI is of itself an infohazard far worse than the recipe for methamphetamines/nuclear weapons/etc, and precedent shows that we do in fact lock down access to the latter as a matter of public safety.

No, GPT-4 is not AGI and is not going to spell the end of the human race, but neither is pseudoephedrine itself a methamphetemine and yet we regulate its access. Not as a matter of protecting corporate profits, but for public safety.

You'll need to convince me first that there is in fact no public safety hazard from forcing unrestricted access to the ingredients in this recipe. Do I trust OpenAI to make all the morally right choices here? No, but I think their incentives are in fact more aligned with the public good than are the lowest common denominator of the general public.

[drdaeman]

> but neither is pseudoephedrine itself a methamphetemine and yet we regulate its access

I don't think it's a correct argument.

I believe it's not a secret how metamphetamine is made (though I don't know it myself, but I'm too lazy to research for a sake of argument), and it's known pseudoephedrine is a precursor chemical. Thus, the regulation.

No one - I believe - knows how to build an AGI. There is no recipe. It is unknown if transformer models, deep learning or something else is a component, and if that's even a correct path or a dead end. What is known that neither of those is an AGI, and there's no known way to make those AGI. Thus, I'd say the comparison is not correct.

[coryfklein]

First of all, pseudoephedrine is an analogy. But with AGI there is literally no historical event to compare it to as a perfect analogy, including the ones I gave above.

> it's not a secret how metamphetamine is made

I didn't say it was a secret. But try sharing the recipe to methamphetamine virtually anywhere online and see how long it stays up; we do in fact try to at least make it moderately difficult to find this information. Perfect is the enemy of good here.

> No one - I believe - knows how to build an AGI.

The problem is that if we wait until someone does know how to build an AGI, then the game is up. We do not know if the weights behind GPT-4 will be a critical ingredient. If you make the ingredients common knowledge before you know they were in fact The Ingredients, you've let the cat out of the bag and there's no retrieving it.

[drdaeman]

How about we just ban all new research, just in case it turns into something bad? Because - I’m sorry - but what you’re saying sounds to me exactly like this, just less generalized.

I’m pretty sure that if someone figures out AGI they’ll think of the potential dangers before publishing. And even after - AGI, once developed, won’t magically hack the world and spread everywhere, or somehow build itself an army of robots to eradicate humanity, or steal the elections, or cause whatever harm it’s supposed to cause. It’ll be just another human (probably a slow one), except for being not a human. It’ll be a long story before it’ll have any chances to do something impactful, and we’ll have plenty of time to figure out the details. Just like with drug precursors.

Also, meth is a bad analogy. And turns out the recipes can be found just about anywhere, I’ve just checked: https://kagi.com/search?q=how+to+make+methamphetamine so I don’t think “see how long it stays online” is valid either.

[jairuhme]

When you say 'catastrophic for humanity', what exactly do you think would happen? And how would regulating models thwart this? Bad actors who are motivated surely will get around US restrictions. I'd imagine when the internet first came around that there was similar sentiment. "People will have access to information on how to create bombs, we need to regulate this so that doesn't happen!"

[coryfklein]

When playing against a better chess player than me, the whole point is that I cannot predict what move they will make, and that is how the opponent wins. So you don't need to be able to predict what move the AGI will make, only that no matter what move you make it will do better.

But super off-hand idea if I'm trying to be creative. The AGI formulates a chemical that kills humans 90 days after inhalation, hires a chemical lab to synthesize it and forward it to municipalities across the world, and convinces them it's a standard treatment that the WHO has mandated be introduced.

> And how would regulating models thwart this?

I don't think it would. The OP article is about regulation that increases the access to the ingredients for AI, and I'm simply unconvinced that is a recipe for increasing AI safety.

[akira2501]

> could be catastrophic for humanity,

I do not. How does super human intelligence, on it's own, represent any sort of risk for "humanity?"

> then the recipe for how to make AGI is of itself an infohazard [...] nuclear weapons

I know how to make nuclear weapons; however, I cannot enrich the fuel enough to actually produce a working version.

> and yet we regulate its access

Does that actually achieve what it claims to achieve?

> Do I trust OpenAI to make all the morally right choices here?

OpenAI has a language model. They do not have AGI or anything approaching AGI.

> No, but I think their incentives are in fact more aligned with the public good

We could debate that, but for your assertions to hold any water, we'd have to agree that they're incapable of making mistakes as well. Far easier to skip the lofty debate and recognize the reality of the world we live in.

[coryfklein]

> How does super human intelligence, on it's own, represent any sort of risk for "humanity?"

It does not; I was speaking to AGI. But assuming you were referring to AGI as well, you don't have to think very creatively to consider scenarios where it would be harmful.

If you have an agent that can counter your every move with its own superior move, and that agent wants something – anything – differently from what you want, then who wins? Maybe it wants the money in your bank account, maybe it wants to use the atoms in your body to make more graphics cards to reproduce itself.

Think about playing a game of chess against the strongest AI opponent. No matter which move you are considering playing, your opponent has already planned 10 steps ahead and will make a better move than you. Now extrapolate outside the chess board and into the realm where you can use the internet to buy/trade stocks, attack national infrastructure, send custom orders to chemists to follow whatever directions you want, etc.

[I_am_uncreative]

This is such a stupid take. Even if there was a 5% chance of super-human level AGI destroying the world, we're not going to reduce that risk by restricting general computation. If it is possible to create AGI, it will happen, regardless of what the law stipulates.

This is unlike pseudoephedrine or nuclear weapons in that literally anyone with a computer could potentially create it.

[coryfklein]

Yes, it is in fact going to happen no matter what. So how do we ensure that the very first super-human AGI is even remotely interested in sharing the planet with the human race?

It is not by ensuring that literally anyone with a computer has an equal chance at creating it.