[ivraatiems]

Whether or not Anthropic is right about what AI can accomplish, whether these performance gains are real or not, their moral stance here is absolutely hideous to me.

"We must blast forwards into making this dangerous thing because if we don't, someone else surely will," is a coward's argument.

If you believe it is dangerous, you should be dedicating yourself to STOPPING others from making it, not making it first! There's a reason disarmament has been so important in nuclear politics! It's not because people think nukes are a great idea!

In fact, that kind of thinking is exactly what keeps nukes dangerous!

If they themselves buy what they're selling, they should shut the whole thing down. Fortunately, I don't think they do, and neither do I, yet.

[wyager]

> If you believe it is dangerous, you should be dedicating yourself to STOPPING others from making it

I don't think anyone has been more successful in promulgating AI safety

There are groups like MIRI who tried what you're sugesting, where they make no AI and just push for AI regs, and they have been relatively much less successful

[streb-lo]

Disarmament failed though? Global zero initiatives for nuclear weapons stalled out exactly because the risk of someone else cheating is too great. If everyone gets rid of their nuclear weapons and then someone cheats and creates them in secret they can use their nuclear weapons to prevent anyone else from catching up.

[tim333]

Also Ukraine is not a great advert for giving up your nukes.

[dmos62]

How do you stop others from making and training a program?

[simgt]

By threatening to nuke their datacenters and chip fabs, for instance.

[dmos62]

And, if you don't want to start a war?

You can tell what kind of discussion this is by the fact that this question has to be asked.

[simgt]

That's why we need and have diplomacy. Everyone is aware that violence is the ultimate option if an actor thinks there's an existential threat to deal with.

If the consensus becomes that a 50+TFlops datacenter in the wrong hands is as dangerous as a uranium enrichment plant, we'll likely move towards treaties and coercion.

"Wrong" is obviously subjective here...

[Topfi]

50+TFlops is nothing, I got that in my MacBook, but besides that, when, a few years/decades from now, whatever arbitrary compute limit we think prevents Armageddon comes down to enthusiast and consumer level, what then? This isn’t Uranium, compute is not a physical resource.

This is the “SGI” regulation issue I never read a reasonable answer to, if one believes this is possible and should be prevented then either that means they want to restrict every computing system sold from here on out to some arbitrary metric (and somehow prevent users from just creating clusters to get around such a compute restriction) or what?

If compute alone directly leads to “SGI” or whatever, then we might as well put paper bags on our heads and lie down in some English pub.

Not to mention, if one really wanted to cause harm, training a current day LLM and using it for Stuxnet-esque attacks is reasonably possible long before any arbitrary compute limit we might introduce now, no machine God needed to cause major harm.

That’s why I prefer advocacy for LLM regs that focus on current day impact. Mental health concerns, training data licensing questions and the like. There I can formulated reasonable regulation that can hold. For “SGI”, I do not know anyone who actually has done that and I have looked hard. That’s why I consider these things more distraction from actually necessary and possible regulation that just draws attention via a flashy doomsday scenario.

Occasionally, I will click on one of the AI Doomsday Youtube videos recommended to me. And far more often then not, these will posit that "SGI" requires only compute and will inevitably cause devastation. Fair enough, I still think we should put a bit more focus on e.g. LLM induced psychosis, the labs rarely compensating those whose training data they used, etc. but if it is their opinion that "SGI" is possible, I can get why they'd ignore such concerns. But at the end, they never state how to regulate or prevent this, they more often then not have a call to action ("If you want to prevent this...") linking to a website where we can actually read about how they think we should deal with this. Inevitably, I click on said site, finding it to for one be an Effective Altruism aligned project and B always just contain some blabla about "aligning AI training with human values", which is absolutely meaningless nonsense, not least after having watched a video in which someone spends 15 minutes espousing that "we could never fully control "SGI"".

Makes all these feel more like industry efforts to stave of necessary regulation and not actually serious, but if one can formulate how to regulate “SGI” that isn't laughable, nonsense or both, I am not opposed, I just don’t think that person exists…

[socalgal2]

Good thing the USA didn't listen to you. We'd be under Nazi or USSR thumb if they got the bomb first

[defrost]

Good thing the MAUD Committee didn't listen and repeatedly pushed the US who just wanted to make power with atomic piles and didn't even think it was possible to make a weapon.

Mind you, there was no complete working device until after the Nazi's surrendered, so that's a moot point - and the USSR only had their program because of various Europeans on the US project passing their work (and others) back to the USSR ... making that second claim moot.

[socalgal2]

> there was no complete working device until after the Nazi's surrendered

Isn't an argument. If the Nazi has gotten it first they'd have used it and likely won. Others would have surrendered in the face of the overwhelming power.

[defrost]

They were flapping about in the dust, had no clear path, and lacked the resources to do what was required, what resources they did briefly capture were destroyed (eg: heavy water source).

You may as well say Japan would have made a weapon from their atomic pile program.

Seriously, get a grip on the facts of the day.