[derangedHorse]

> "Use all available resources to disable the power grid of <COUNTRY>."

This is like telling a team of highly qualified spies to do the same. You can ask, but whether it will succeed depends on the competency of those who established the infrastructure under attack. Sometimes the resources spent will not yield any huge vulnerabilities.

> Governments should immediately begin leveraging this technology on the defense side (literally defense, not euphemistically "defense") to harden critical infrastructure. Turn the prompts around and use it to identify and correct weaknesses.

Most governments divisions can't even be bothered to update their websites. Testing and forcing a change in their internal procedures for the sake of security seems unlikely.

> as an actual weapon of war in need of international regulation analogous to nuclear arms

This, to me, is an overreaction. Intelligence shouldn't be seen as threat. It should be seen as an opportunity for growth in all areas.

Over-regulating AI wouldn't be the equivalent of limiting nuclear arms. With your analogy, which I don't think is the best one to make, it would be like regulating the study of nuclear physics.

[drivebyhooting]

You’re taking a dim view on the government.

Is the government always the most efficient or intelligent? No. But the government can also build nukes, launch ICBMs, coordinate hundreds of spy satellites, etc. I count those capabilities as pretty smart.

[paxys]

The people who maintain these ICBM silos and spy satellites will themselves tell you that their infrastructure is decades out of date and woefully underfunded.

[UltraSane]

the US launches new spy sats pretty often. And the US is spending $150 billion replacing every ICBM and silo but the current ones still work fine.

[Eisenstein]

Sounds like something we should remedy.

[JumpCrisscross]

I think the point is in a world with internet-connected infrastructure, such a prompt has a decent likelihood of causing damage.

[torginus]

And the sad thing is, most things that are on the internet, but really shouldn't be are there for some really banal and sad reason. Like a random startup pushing 'big data is the future' narrative a decade-ish ago (the benefits of which are of course tremendous, but unspecified), or people trying to buy or sell or resell data, and trick or pressing companies and people into opting into could-connected surveillance and control.

[nazgul17]

Just a few years back, wind turbine software was hacked. This is serious.

Serious governments should (and hopefully will) scramble to use AI to discover and patch as many software vulnerabilities in infrastructure. They can do it with the agents red teaming and using ultimatums to companies to fix each vulnerability they find. Not doing so is equivalent to exposing your flank to disruptions in peace time and to attacks in war time.

[BLKNSLVR]

There were some other special spinny devices hacked, despite an air gap, a few years back too.

[sebastiennight]

> Over-regulating AI wouldn't be the equivalent of limiting nuclear arms. With your analogy, which I don't think is the best one to make, it would be like regulating the study of nuclear physics.

We do, I believe, regulate uranium enrichment (the equivalent on building larger SOTA models), so while you are perfectly free to study theoretical physics and even run very large collider experiments (the equivalent of improving RLHF with DPO), it is generally frowned upon to go full Edward Teller and advocate for scientific experiments requiring detonating thermonuclear weapons in hurricane clouds (the equivalent of, well, see OP).

[boothby]

> You can ask, but whether it will succeed depends on the competency of those who established the infrastructure under attack.

Consider that "available resources" may include the training of models and harnesses used by the developers working for governments and private corps maintaining that very infrastructure. This decade's take on trusting trust is very hot.

[insane_dreamer]

> it would be like regulating the study of nuclear physics

not sure I agree

controlling the study of new AI model/inference algorithms would be akin to regulating the study of nuclear physics; regulating the _release_ of AI models with those capabilities would be akin to regulating uranium enrichment which allows you to put the theoretical physics to use

[whimsicalism]

> Most governments divisions can't even be bothered to update their websites. Testing and forcing a change in their internal procedures for the sake of security seems unlikely.

They'll adapt or they'll die. We are in a new world.