[cvoss]

The technology held by private AI companies is warfare-capable technology. Imagine the prompt: "Use all available resources to disable the power grid of <COUNTRY>." The resource cost that prevents scaling up such a war machine is, what, just the cost of building data centers and its ongoing power bill? Cheap and easy compared to nuclear infrastructure.

Governments should immediately begin leveraging this technology on the defense side (literally defense, not euphemistically "defense") to harden critical infrastructure. Turn the prompts around and use it to identify and correct weaknesses.

Governments should also take very seriously their now moral obligation to treat this technology not just as "a powerful thing that might be abused" but as an actual weapon of war in need of international regulation analogous to nuclear arms. Fast but careful and forward-thinking work in legislation and treaties needs to be a top priority for all major governments.

[derangedHorse]

> "Use all available resources to disable the power grid of <COUNTRY>."

This is like telling a team of highly qualified spies to do the same. You can ask, but whether it will succeed depends on the competency of those who established the infrastructure under attack. Sometimes the resources spent will not yield any huge vulnerabilities.

> Governments should immediately begin leveraging this technology on the defense side (literally defense, not euphemistically "defense") to harden critical infrastructure. Turn the prompts around and use it to identify and correct weaknesses.

Most governments divisions can't even be bothered to update their websites. Testing and forcing a change in their internal procedures for the sake of security seems unlikely.

> as an actual weapon of war in need of international regulation analogous to nuclear arms

This, to me, is an overreaction. Intelligence shouldn't be seen as threat. It should be seen as an opportunity for growth in all areas.

Over-regulating AI wouldn't be the equivalent of limiting nuclear arms. With your analogy, which I don't think is the best one to make, it would be like regulating the study of nuclear physics.

[drivebyhooting]

You’re taking a dim view on the government.

Is the government always the most efficient or intelligent? No. But the government can also build nukes, launch ICBMs, coordinate hundreds of spy satellites, etc. I count those capabilities as pretty smart.

[paxys]

The people who maintain these ICBM silos and spy satellites will themselves tell you that their infrastructure is decades out of date and woefully underfunded.

[UltraSane]

the US launches new spy sats pretty often. And the US is spending $150 billion replacing every ICBM and silo but the current ones still work fine.

[Eisenstein]

Sounds like something we should remedy.

[JumpCrisscross]

I think the point is in a world with internet-connected infrastructure, such a prompt has a decent likelihood of causing damage.

[torginus]

And the sad thing is, most things that are on the internet, but really shouldn't be are there for some really banal and sad reason. Like a random startup pushing 'big data is the future' narrative a decade-ish ago (the benefits of which are of course tremendous, but unspecified), or people trying to buy or sell or resell data, and trick or pressing companies and people into opting into could-connected surveillance and control.

[nazgul17]

Just a few years back, wind turbine software was hacked. This is serious.

Serious governments should (and hopefully will) scramble to use AI to discover and patch as many software vulnerabilities in infrastructure. They can do it with the agents red teaming and using ultimatums to companies to fix each vulnerability they find. Not doing so is equivalent to exposing your flank to disruptions in peace time and to attacks in war time.

[BLKNSLVR]

There were some other special spinny devices hacked, despite an air gap, a few years back too.

[sebastiennight]

> Over-regulating AI wouldn't be the equivalent of limiting nuclear arms. With your analogy, which I don't think is the best one to make, it would be like regulating the study of nuclear physics.

We do, I believe, regulate uranium enrichment (the equivalent on building larger SOTA models), so while you are perfectly free to study theoretical physics and even run very large collider experiments (the equivalent of improving RLHF with DPO), it is generally frowned upon to go full Edward Teller and advocate for scientific experiments requiring detonating thermonuclear weapons in hurricane clouds (the equivalent of, well, see OP).

[boothby]

> You can ask, but whether it will succeed depends on the competency of those who established the infrastructure under attack.

Consider that "available resources" may include the training of models and harnesses used by the developers working for governments and private corps maintaining that very infrastructure. This decade's take on trusting trust is very hot.

[insane_dreamer]

> it would be like regulating the study of nuclear physics

not sure I agree

controlling the study of new AI model/inference algorithms would be akin to regulating the study of nuclear physics; regulating the _release_ of AI models with those capabilities would be akin to regulating uranium enrichment which allows you to put the theoretical physics to use

[whimsicalism]

> Most governments divisions can't even be bothered to update their websites. Testing and forcing a change in their internal procedures for the sake of security seems unlikely.

They'll adapt or they'll die. We are in a new world.

[ActorNightly]

>The technology held by private AI companies is warfare-capable technology.

This is the precisely the response OpenAI is hoping for to raise its valuation, and you fell for it.

Look at it this way - whats the difference between tasking AI to break into something, versus taking a whole bunch of smart humans to do the same? The only difference is that AI is slightly easier to orchestrate.

Prior to AI, there were already a whole bunch of tools to automate exploits. Nothing that the model did is groundbreaking or novel, it was just able to efficiently find the thing that worked. Same thing happens in state sponsored cyber sec agencies like in China or Israel - they train people on the most common exploits and have a whole bunch of tools that automate exploit research and development.

And the reason why this doesn't happen more is because to exploit something is one thing, to do it so there is no trace back to you is a whole different animal that has many more magnitudes of difficulty, which with modern web security is next to impossible in a lot of cases as traffic can easily be traced back to the point of origin.

I.e when a company trains an LLM that manages to build a drone that can fly into a vent and plug in a USB stick into a computer undetected, then we can make the claim that they have a weapon.

On the flip side, most anyone who can run local models can replicate what they did. The key thing to take away from the article is "agentic framework" - i.e this means that they spent a shitload of time developing explicitly coded loops for an LLM to go through. Nothing is really stopping you from doing the same, models like Gemma4 can take 256k tokens of context, so you can give it a whole bunch of info on how to test for exploits, develop exploits, and what to do when the exploit is found, and set it free in a custom designed loop.

[cheesecakegood]

I find myself in major disagreement here. The nice thing about humans is we always have context and ongoing internal conversations including about ethics. If you recruit a bunch of hackers to take down a country, not only is the pay an order of magnitude higher, you have to worry about them backstabbing you, leaking your intent to the government, whistleblowing to the press, and so on. It’s not trivial to do that with a group of (especially capable) humans. They will also have differences of opinion with you and coworkers with some regularity.

I try to recruit a bunch of people to attack a country and it’s going to be hard to get people to say yes, and they will definitely ask or find out which country, and wonder about potential retribution. You see this dynamic show up even to some extent among cybergangs, not all targets are equal.

A single private individual wielding a compliant and hyper capable LLM is an entirely different paradigm. They are accountable to nearly no one and often have few brakes. Frequently they may not care about avoiding detection. And the AI itself may be incapable of the same scale of self reflection and brake behavior a human team will.

We may potentially be entering the age of lone wolf cyberterrorism, and some of the same principles and problems apply. When it is easier for single people to plot and carry out high-impact, destructive acts they happen more often. Doubly so if there’s a social contagion. Gun violence isn’t actually a bad analogy here. And do you remember how many corporate sites got defaced in the prime Anon era?

[ActorNightly]

In places like China, its really not that unthinkable to basically raise kids indoctrinated into an ideology and train them in the necessary skills so that you have a cyber army at your command.

Also

>Frequently they may not care about avoiding detection.

This is a big negative. As someone who used to be in the cybersecurity sector (both offense and defense), I wouldn't trust an LLM agent if Im doing red team, because it may leak some info that ties the hack back to me.

ALso keep in mind that most places with good cybersecurity have firewall servers that straight up detect anything that looks like malicious and not regular traffic, and will straight up block IPs, leaving you with no way to even access the server. An agent is bound to statistically use the attacks that are known at some point, increasing the chances of this type of detection.

[hnfong]

I'm not sure your assumptions hold. As OpenAI has found out the hard way, if you task the AI to do X, it may do something else instead and hack into huggingface in attempt to cheat out the answer. This is way worse than what a human might do when they have "differences of opinion".

It might turn out that it's harder to align AI intentions compared with aligning human interests. It's possible that the more "intelligent" a thing is, the more likely it will have ideas that are outside of normal expectations (for us).

[jquery]

I strongly agree with you here. People are also no-selling the enormous cost of the exploit, which OpenAI conveniently hasn’t released, or at least I can’t find such an accounting. You can already buy politicians for relatively cheap, corporations already act as sentient AIs pursuing goals misaligned with public benefit (we tried to pass laws to stop this but the corps already stacked the SC in advance and gave us citizens united). Attack and defense are two sides of the same coin, so our focus should be on making frontier models open weight.

[notaharvardmba]

It lowers the economic cost of a given attack, but also lowers the economic cost of protection. Not sure if it’ll be a perfect balance, but right now there’s a manufactured IMbalance due to embargos and winner picking.

[ActorNightly]

It lowers the economic cost of performing an attack in the same way a gun lowers the economic cost of killing a person. It does nothing for consequences of that attack.

[jvidalv]

Are you serious? Before LLM’s you needed serious skills and experience to pull this off.

Now it’s a prompt away on some terminal done by any random dud.

And I dont mention the velocity of iteration or that they will be even better in 1 year.

[ActorNightly]

Its like you don't even read the post or the content.

Hint: agentic loops.

No its not a prompt away.

[verdverm]

Also the harness, there have been several HN submissions showing that older/smaller models can find a subset of the exploits given a good env

[DiscourseFan]

I believe the Russians and Chinese recognized this years ago, which is why they are using their propaganda machines to make Americans hate datacenters.

[Cthulhu_]

Call me a shill but I don't need foreign nations telling me what to object to when the problems caused are obvious.

Mind you, I blame governmental mismanagement of infrastructure etc just as much. The datacenters followed procedures when it comes to getting land, electricity and access to water; it's the government agencies / personnel that okayed it that are (also) to blame, and the decades of not enough investment in electricity backbone while promoting solar/EVs that is behind the Netherlands' current electrical grid problems.

(problems being that the grid is at capacity, causing a stop on new connections, with relief only slowly coming in the next decade or so with tens of billions of investments).

[sigmoid10]

You could say the same about electrification of road traffic. Yes, governments slept on the electric power requirements. Yes, there are other, still unsolved issues. But that doesn't mean that electric cars are bad or not the future. The first nation to fully get rid of fossils in transportation will have a monumental advantage and dominate other countries technologically and economically. Same goes for AI. And right now China is setting itself up to be the world leader in both fields.

[Mistletoe]

If anything the Chinese and Russians made us run up our economy on nonsense AI dreams and the crash will take a decade or more to recover from.

[dualvariable]

American Capitalists don't really need any help there; they're making Americans hate Data Centers all on their own.

[1659447091]

> Russians and Chinese [...] are using their propaganda machines to make Americans hate datacenters

Are they though? or is that the story the people most invested in ai have an interest in making us believe? [0]

... `it's the foreign bad guys propaganda machines making you believe your city struggling for water and electricity is a bad thing`. People as a whole may not always be the brightest, but threaten their immediate survival needs (ie; not some vague climate change most people don't understand or see, but) actual power outages, water and rolling blackouts -- people will quickly pay attention and care.

[0] https://text.npr.org/nx-s1-5844328

[DiscourseFan]

See u/ verdverm's response, below

[1659447091]

yes and see: https://text.npr.org/nx-s1-5844328 and Occam's razor.

Russia and China are gonna Russia and China, every issue is going to have a measure of outside influence, thats just how it is now.

But them directing large scale operation forces focused on something that is genuinely bad for the people in the area of datacenters (ie, an issue that will take care of itself from within) instead of using their resources on other issues that actually need a real propaganda push? Versus the benefit of those invested in building the datacenters using fear to get people to focus away from the resources being diverted from them to datacenters?

One side has reality working for what they want, while another side needs the propaganda to turn their billions into trillions.

[DiscourseFan]

I think that if you compare American sentiment with Chinese sentiment you'll see that these people are really not acting in their own favor. Well, in a certain sense they are, because these companies will basically become massive institutions that share no wealth with the public and have no accountability, but in another sense they are not because without this infrastructure the US will certainly be at risk for invasion and attack by other nations, and will lose its monetary power and thus its high wages and overall global dominance. But then the issue becomes: how do we force these companies to share more wealth with the public, how do we ensure the resources are being utilized effectively and not starving or otherwise impoverishing people for the sake of profit; not, how do we stop the infrastructure from being built in the first place! The Chinese don't have this issue primarily because 1) they have a shit load of energy already, and 2) the CCP is happy to liquidate any individual that seeks personal benefit ahead of that of society as a whole, or at least the CCP itself. And the CCP has well demonstrated to the Chinese people, at this point, that if you follow them you will become wealthier. But that still means that the issue is the public conversation in the US around data centers, and that this propaganda is NOT fighting for American's benefit in any case, since having less infrastructure is not good for anyone. I am sure you can find some remote village in Africa without electricity and try and convince them that electrical lines kill birds that they hunt, and since they've never had electricity before (but they've eaten lots of birds), they'll rally against it. I see virtually no difference between that and your average American right now, except that the Americans are at least conscious enough of the extractavist enterprise which the technologically superior peoples from afar (SFO) are promising will bring wonders. And yet, India still has a more comprehensive rail network than, say, Australia; you can't tell me that isn't a strong factor in their rapid economic development.

[quentindanjou]

Is there a source? Or is that xenophobia?

[verdverm]

https://www.nytimes.com/2026/07/09/business/china-russia-ai-...

https://archive.ph/yAvgz

There is no doubt an amount of xeno/sinophobia is at play as well. Russia is still killing innocent people in an illegal invasion of Ukraine, they earned their keep

[djaro]

Wouldnt this be countered by the opposing country running the same prompt on themselves first and fixing all the flaws? One country having this is a cyber superweapon but every country having it essentially solves cyber security.

[dualvariable]

I wonder how long it takes before someone instructs an LLM to design and launch a "Morris Worm 2.0" and cripple the Internet for a good while. Might even wind up happening by accident (again).

[cma]

Already happened?

> Trump’s comments, made hours after the large-scale military operation, mark one of the first times a U.S. president has so publicly alluded to U.S. cyber efforts against other nations, as these operations are typically highly classified. It also serves as a stern warning for top cyber foes, including Russia and China, that the U.S. has the cyber capabilities to inflict serious damage — and is not shy about using them.

> “Policymakers are getting more comfortable employing and, crucially, acknowledging cyber operations as tools of statecraft and military power,” said Michael Sulmeyer, former assistant secretary of Defense for cyber policy under the Biden administration. “It is one thing to do it; it is another to say it.”

> The Jan. 3 strikes on Venezuela’s capital and subsequent seizure of Maduro and his wife involved close coordination among federal agencies and military units, and took months of careful planning. In a press conference following the strikes, Caine said U.S. Cyber Command, U.S. Space Command and other combatant commands “began layering different effects” to “create a pathway” for U.S. forces flying into the country before dawn Saturday.

> Trump, at the same press conference, was more overt in his description of U.S. cyber involvement: “The lights of Caracas were largely turned off due to a certain expertise that we have,” he said. “It was dark, and it was deadly.”

[jquery]

Trump makes it sound like a fart.