[Enginerrrd]

> it simply does not require much intelligence (relatively speaking) to build something that points a gun at something and pulls a trigger?

I could not disagree more. A big part of that is also knowing when NOT to pull the trigger. And it’s much harder than you’d think. If you think full self driving is a difficult task for computers, battlefield operations are an order of magnitude more complex, at least.

[ACCount37]

We have fully autonomous weapons, and had them for over a century. We call them "landmines".

I expect autonomous weapons of the near future to look somewhat similar to that. They get deployed to an area, attack anything that looks remotely like a target there for a given time, then stand down and return to base. That's it.

The job of the autonomous weapon platform isn't telling friend from foe - it's disposing of every target within a geofence when ordered to do so.

[fweimer]

And the arms industry has been pushing smart mines for decades, so that they can keep selling them despite the really bad long-term consequences (well beyond the end of hostilities) and the Ottawa Treaty ban. In the end, land mines are killing people although the mines are supposed to be sufficiently advanced not to target persons.

From a security perspective, the “return to base” part seems rather problematic. I doubt you'd want to these things to be concentrated in a single place. And I expect that the long-term problems will be rather similar to mines, even if the electronics are non-operational after a while.

[ACCount37]

"Smart mines" specifically can be designed so that they're literally incapable of exploding once a deployment timer expires, or a fixed design time limit is reached.

It just makes the mines themselves more expensive - and landmines are very much a "cheap and cheerful" product.

For most autonomous weapons, the situation is even more favorable. Very few things can pack the power to sit for decades waiting for a chance to strike. Dumb landmines only get there by the virtue of being powered by the enemy.

[golem14]

Well, I assume that they are at least not to attack their autonomous "comrades". Masquerading as such will be one obvious tactic, no ? You could argue that these guys would use e2e encrypted messages as FOF designation, but I would imagine a contested area would be blanketed with jammers, leaving only other options (light ? but smokescreens. Audio? Also easily jammed). So this isn't as easy as most people think.

Edit: No, I don't think a purely defensive stance like landmines is sufficient and what the people in command think.

We have landmines today. Why spend much more making marginally better, highly intelligent ones with LLMs?

[golem14]

Also, a longer quote from Douglas Adams might be appropriate here (also appropriate to agentic vibe coding ...)

Click, hum.

The huge grey Grebulon reconnaissance ship moved silently through the black void. It was travelling at fabulous, breathtaking speed, yet appeared, against the glimmering background of a billion distant stars to be moving not at all. It was just one dark speck frozen against an infinite granularity of brilliant night. On board the ship, everything was as it had been for millennia, deeply dark and Silent.

Click, hum.

At least, almost everything.

Click, click, hum.

Click, hum, click, hum, click, hum.

Click, click, click, click, click, hum.

Hmmm.

A low level supervising program woke up a slightly higher level supervising program deep in the ship's semi-somnolent cyberbrain and reported to it that whenever it went click all it got was a hum.

The higher level supervising program asked it what it was supposed to get, and the low level supervising program said that it couldn't remember exactly, but thought it was probably more of a sort of distant satisfied sigh, wasn't it? It didn't know what this hum was. Click, hum, click, hum. That was all it was getting. The higher level supervising program considered this and didn't like it. It asked the low level supervising program what exactly it was supervising and the low level supervising program said it couldn't remember that either, just that it was something that was meant to go click, sigh every ten years or so, which usually happened without fail. It had tried to consult its error look-up table but couldn't find it, which was why it had alerted the higher level supervising program to the problem.

The higher level supervising program went to consult one of its own look-up tables to find out what the low level supervising program was meant to be supervising.

It couldn't find the look-up table.

Odd.

It looked again. All it got was an error message. It tried to look up the error message in its error message look-up table and couldn't find that either. It allowed a couple of nanoseconds to go by while it went through all this again. Then it woke up its sector function supervisor.

The sector function supervisor hit immediate problems. It called its supervising agent which hit problems too. Within a few millionths of a second virtual circuits that had lain dormant, some for years, some for centuries, were flaring into life throughout the ship. Something, somewhere, had gone terribly wrong, but none of the supervising programs could tell what it was. At every level, vital instructions were missing, and the instructions about what to do in the event of discovering that vital instructions were missing, were also missing. Small modules of software - agents - surged through the logical pathways, grouping, consulting, re-grouping. They quickly established that the ship's memory, all the way back to its central mission module, was in tatters. No amount of interrogation could determine what it was that had happened. Even the central mission module itself seemed to be damaged.

This made the whole problem very simple to deal with. Replace the central mission module. There was another one, a backup, an exact duplicate of the original. It had to be physically replaced because, for safety reasons, there was no link whatsoever between the original and its backup. Once the central mission module was replaced it could itself supervise the reconstruction of the rest of the system in every detail, and all would be well.

Robots were instructed to bring the backup central mission module from the shielded strong room, where they guarded it, to the ship's logic chamber for installation.

This involved the lengthy exchange of emergency codes and protocols as the robots interrogated the agents as to the authenticity of the instructions. At last the robots were satisfied that all procedures were correct. They unpacked the backup central mission module from its storage housing, carried it out of the storage chamber, fell out of the ship and went spinning off into the void.

This provided the first major clue as to what it was that was wrong.

[snowwrestler]

You don’t need Anthropic for this use case, so obviously this use case is not what the current fight is about.

[ACCount37]

You don't need Anthropic for any use case. They don't ship VLAs either - nothing from Anthropic's entire model lineup can run on a killer drone.

Which raises the question: why did the Pentagon try to pressure Anthropic at all?

On the principle of it? Political reasons? Or was the real concern "domestic warrantless surveillance"?

[mothballed]

I guess by that definition, a bullet is also autonomous. It will strike anything in its path of flight, autonomously without further direction from the operator.

[disillusioned]

Bullets don't kill people, etc. etc.

If anything represents the logical conclusion of that tired fallacy, it'll be actually autonomous, "thinking" drones which make the targeting decisions and execution decisions on their own, not based on any direct, human-led orders, but derived from second-order effects of their neural net. At a certain point, it's not going to matter who launched the drones, or even who wrote the software that runs on the drones. If we're letting the drones decide things, it'll just be up to the drones, and I don't love our chances making our case to them.

[strangattractor]

"Since the end of the Vietnam War in 1975, unexploded ordnance (UXO)—including landmines, cluster bombs, and artillery shells—has killed over 40,000 people and injured or maimed more than 60,000 others." - Google AI Overview "How many children were maimed by landmines after the vietnam war"

[thejohnconway]

Yes, but it doesn’t have to be error-free. The friendly fire rates in symmetrical hot wars is pretty high, it’s considered a cost of going to war.

If autonomous weapons lead to a net battlefield advantage, I agree with the GP, they will be used. It is the endgame.

[collingreen]

The big asterisk in what you're saying is, like self driving cars, it's hardest when you want to be the most precise and limit the downsides. In this paradigm, false positives and false negatives have a very big cost.

If you simply wanted to cause havoc and destruction with no regard for collateral damage then the problem space is much more simple since you only need enough true positives to be effective at your mission.

The ability to code with ai has shown that it requires an even higher level of responsibility and discipline than before in order to get good results without out of control downside. I think the ability to kill with ai would be the same way but even more severe.

[davidw]

> A big part of that is also knowing when NOT to pull the trigger

"In a press conference, Musk promised that the Optimus Warbots would actually, definitely, for real, be fully autonomous in two years, in 2031. He also extended his condolences to the 56 service members killed during the training exercise"

[ben_w]

I've not watched all of Robocop (too much gore for me), but I have seen the boardroom introduction of the ED-209.

That's how I imagine a Musk demo of this kind of thing would play out, if his team can't successfully manage upwards.

[0xffff2]

And the US learned the lesson the hard way in Iraq that in fact even human intelligence struggles with this. There were major problems throughout the war with individual soldiers not adhering to the published rules of engagement.

[NoGravitas]

Yes, but the important bit is that autonomous drones can't be held accountable for not adhering to the published rules of engagement.