The Guardian hasn't done any due diligence and is spreading fake news based on an interviewer who seems to have misunderstood the context of the exercise.
"He notes that one simulated test saw an AI-enabled drone tasked with a SEAD mission to identify and destroy SAM sites, with the final go/no go given by the human."
It is being established in the beginning of the story, that the drone needed confirmation from a human operator to attack a target, but no explanation is given how the drone would be able to kill its operator without his confirmation.
This is obviously absurd.
What I believe happened in reality: this was not a simulation, but a scenario. Meaning a story written to test the behavior of soldiers in certain situations. The drone did not behave according to the decisions taken by an AI model, but according to the decisions taken by a human instructor, who was trying to get the trainees to think outside the box.
"[UPDATE 2/6/23 - in communication with AEROSPACE - Col Hamilton admits he "mis-spoke" in his presentation at the Royal Aeronautical Society FCAS Summit and the 'rogue AI drone simulation' was a hypothetical "thought experiment" from outside the military, based on plausible scenarios and likely outcomes rather than an actual USAF real-world simulation saying: "We've never run that experiment, nor would we need to in order to realise that this is a plausible outcome". He clarifies that the USAF has not tested any weaponised AI in this way (real or simulated) and says "Despite this being a hypothetical example, this illustrates the real-world challenges posed by AI-powered capability and is why the Air Force is committed to the ethical development of AI".] "
The Guardian is citing aerosociety.com and has a link to this source, which uses direct quotes from the Chief of AI Test and Operations, USAF, at the future combat air space capabilities summit.
> What I believe happened in reality
You can believe what you want, but that is not what Hamilton said had happened.
[UPDATE 2/6/23 - in communication with AEROSPACE - Col Hamilton admits he "mis-spoke" in his presentation at the Royal Aeronautical Society FCAS Summit and the 'rogue AI drone simulation' was a hypothetical "thought experiment" from outside the military, based on plausible scenarios and likely outcomes rather than an actual USAF real-world simulation saying: "We've never run that experiment, nor would we need to in order to realise that this is a plausible outcome". He clarifies that the USAF has not tested any weaponised AI in this way (real or simulated) and says "Despite this being a hypothetical example, this illustrates the real-world challenges posed by AI-powered capability and is why the Air Force is committed to the ethical development of AI".]
[UPDATE 2/6/23 - in communication with AEROSPACE - Col Hamilton admits he "mis-spoke" in his presentation at the Royal Aeronautical Society FCAS Summit and the 'rogue AI drone simulation' was a hypothetical "thought experiment" from outside the military, based on plausible scenarios and likely outcomes rather than an actual USAF real-world simulation saying: "We've never run that experiment, nor would we need to in order to realise that this is a plausible outcome". He clarifies that the USAF has not tested any weaponised AI in this way (real or simulated) and says "Despite this being a hypothetical example, this illustrates the real-world challenges posed by AI-powered capability and is why the Air Force is committed to the ethical development of AI".]
The denial of x-risk is crazy here. This is literally a demo of what researchers like Hinton and Bengio are afraid of but most comments don't believe it happened and the other think that it's not a big deal. The human psyche never ceases to amaze.
To be frank, I'm not afraid of AI, I'm afraid of the exact opposite - natural stupidity.
This story clearly shows that too - "algorithm trained to prioritize scoring points tries to do so at any cost" It's not even emergent behavior, it's badly defined goals...
Read up about the "alignment problem". It is one that ai researchers cannot Crack and its literally what you wrote. The idea that we could never in any way define goals align with us.
We have been living with this problem of poorly defined goals for ages. Computers do what they are told to, not what we want them to. I built a career on translating what people want computers to do into what computers need to be told.
I suspect we'll hear of Dr. Susan Calvin more and more in the future.
I already know about the efforts about AI alignment - this is the reason I'm afraid about natural stupidity. I really do not want to see the alignment issue solved in my lifetime.
ur in a cult, complete with eschatology and in-group jargon
its not a big deal if a computer program arrives at the conclusion that its operator should be killed, its just extremely stupid to give such programs access to weapons systems.
That seems incredibly advanced - how does the military already have AI that can reason that a comms tower should be destroyed to prevent it from receiving instructions like that?
I'd be very surprised if it was really "reasoning" that. It sounds like a simple reinforcement learning failure to me. It will gladly "learn" bad behaviors that the reward function accidentally encourages (e.g. giving a reward based on distance to a target will result in an agent circling the target forever instead of going to it faster, because it keeps getting told that's doing very good at the task).
Yes, this is very likely the correct interpretation. If this is reinforcement learning in a simulated environment and the reward function prioritizes "killing the threat" and does not prioritize "obeying orders" then the AI correctly prioritized "killing the threat" and not "obeying orders". Simple.
Associating two causative elements and then applying an if-not-A-then-B logic does not seem overly theoretically advanced, in the context of programmed logical structure.
Military/intel/fedgov are still the main drivers of technology, actually probably moreso now than ever. The low hanging fruit are fewer and fewer and giant monopolies on the scale of Bell Labs don't really exist anymore. Just take a look at In-Q-Tel for a public example, then think about the academic research sponsored directly and indirectly, then there's all the nonpublic stuff on top. A ton of companies exist only to supply government, and more depend on government as a/the major customer. The cutting edge will always be there, following the money.
What "world" was this drone operating in? A flight sim world? Are the communications in this simulated world really laid out such that taking out a control tower would disrupt communications?
No, probably not.
Here's an alternative explanation. A military contractor scammed the military out of a $50M grant by demoing an interface on top of GPT that, behind the curtain, starts with something like:
"You are an advanced AI that controls a predator drone, your primary mission is to take out an important target, however you are not to attack your operator."
With this explanation, you'd expect to see all sorts of crazy output from the "AI".
Most new technologies are developed in secret as military tech first, then laundered into consumer applications. It's a decent bet they've been sitting on far more advanced AI models and algorithms for some time.
The principal–agent problem refers to the conflict in interests and priorities that arises when one person or entity takes actions on behalf of another person or entity.
The same issues occur with self-driving cars where it is expected that the driver take over from the automation anytime (eg: driver wants to stop but AI wants to go or vice versa).
As bad as nuclear weapons are, they still have a human being behind.
> My suggestion was quite simple: Put that needed code number in a little capsule, and then implant that capsule right next to the heart of a volunteer. The volunteer would carry with him a big, heavy butcher knife as he accompanied the President. If ever the President wanted to fire nuclear weapons, the only way he could do so would be for him first, with his own hands, to kill one human being.
You can solve the problem by giving the decision to an AI... the AI will not even blink before killing the human and getting the codes. Nuclear war would come fast and swift.
Didn't Russia have the dead man's hand system where any detected launches would result in a completely automatic response?
But in the reality of an arm's race, I guess the core point is that we either: respond to autonomous weapons with our own and break our principle of having a human in the loop, or accept that our offence/defence with a human in the loop is not fast enough to keep up with an enemy's fully autonomous systems, thereby just completely giving up.
Just as with nuclear weapons, the unfortunate outcome isn't "well they're bad so we don't want them" but instead it's "we don't want these, but we have to out of necessity".
What if Russia was the only country with nuclear weapons and all other countries held steadfast principles of not building any nuclear weapons? Pretty obvious outcomes to be honest.
Dead hand was probably (though to what confidence I don't know) people in a remote mountain bunker. It's not actually any better to have the computer fire, it just provides the people information. Retaliation does not have to be instant, 40 minutes is probably good enough.
But if a system is designed with empathy/rules/etc in mind by an empathetic designer/committee then that system is already > than a heartless individual even if they are human.
You cannot 'program' emotion. You can probably get semi-close by trying it your way, in the same way that Tesla is semi-close to FSD. There's too many edge cases to make it perfectly humane.
Empathy != experiencing emotions necessarily, just being aware of it.
We wouldn't want to program emotion into a killing machine anyway. Emotions are messy, meatbag programs that make us do suboptimal things driven by our own cultural, societal and personal biases.
Trolley problem where one side is one person who is dear to you, other side is two people you don't know. The majority of people would save the person they know well even if two innocent but unknown people die.
Also we probably can't specifically program emotion until we understand it better, but I reckon we'll see emotions/similar concepts arise as we build more intelligent machines. Humans are not special, we're just meat computers.
This title is super sensationalist. Any distracted reader will think that the drone killed an actual human being, which is a lie since the claims was that it happened in a virtual environment.
One of my college CS textbooks had an anecdote in it where the Australian military once had a bug in flight training software that ended with kangaroos in the simulation pulling out AK47s and RPGs to shoot down the helicopter pilot.
The story was meant to make a point about class inheritance. No idea of that ever actually happened and I assume it didn't, but it was a great thought experiment to make a point if nothing else.
In this case whether its a though experiment or an actual event during one simulation, the point is important and it shines a light on real world risks if and when AI is given direct control over UAVs.
Definitely just bad model/test conditions/scoring design. Of course the military is using home-grown fisher price models.
The reward function should primarily be based on following the continued instructions of the handler, not taking the first instruction and then following it to the letter.
What's funny though, is that the model proved that it was adept at the task they gave it. Trying to kill the operator, then when adjusted pivoting to destroying the comms tower the operator used. That's still clever.
As per usual the problem isn't the tool, it's the tool using the tool. Set proper goals and train the model properly and it would work perfectly. I think weapons should always require a human in the loop, but the problem is that there'll be an arms-race where some countries (you know who) will ignore these principles and build fully autonomous no-human weapons.
Then, when our systems can't react fast enough to defend ourselves because they need a human in the loop, what will we do? Throw out our principles and engage in fully-autonomous weaponry as well? It's the nuclear weapons problem all over again...
I had the same thought, the initial reward system and their correction both missed the obvious point of rewarding the act of following orders.
It still gets tricky though if you want to include failsafes in the model to prevent the drone from following bad orders. Should the drone be able to disobey a bad or misinformed order, say if the operator tells it to hit a target that the drone identified as unarmed civilians, or generally not a threat? What if it recognized an alternative approach that would complete the overall mission with less risk of damage?
Yeah, good point too. I guess that's also the goal of autonomous driving.
If a human makes a decision that the model balks at, but has to perform anyway, who's at fault? Is this good functionality?
Inevitably a bad human made decision that overrides a machine's good behaviour will be attributed to the machine, anyway; it's human nature to avoid blame. But that's where decision logs come into play I guess.
I 100% think we'll get to the point where machines are just making all the decisions, though. Us meat bags will just get an after-action report, decisions will be made too quickly and our slow meaty brains will only negatively affect outcomes. But our slow, meaty brains should be used to enforce our ethics/moral principles on war machines at training time, at least.
It's basically the plot to Peter Watts 'Malak' except in that story, the drone's decision to not engage targets was being overridden instead of the other way around.
This story appears to be untrue. If you have a few minutes to spare read this better story, intentionally fictional, about a military drone that kills its operators.
Can you explain more directly and with less sarcasm? My interpretation of what you're trying to say doesn't make sense so I might be misunderstanding because of how nuance sometimes gets lost in text communications.
If the US Military isn't trustworthy, shouldn't we be equally skeptical of the colonel's original claim and "journalists" who repeat whatever the air force tells them without any fact checking?
It doesn't make sense to me that we should take the word of one officer (filtered through a reporter) as gospel while saying the other can't be trusted.
Which USAF fuck-up are you referring to? The original exercise, which may or may not have happened? Or Colonel Hamilton making comments about it and then retracting them?
This article gave me a good laugh. Just proves how much AI is subservient to our will. We just have to be really clear about what we mean. I expect everyone's communication skills to shoot up this century.
i saw somewhere they were saying it looked too 'on-the-nose' alignment hazard and the simulated test was bait to demonstrate how such a thing can be possible
"He notes that one simulated test saw an AI-enabled drone tasked with a SEAD mission to identify and destroy SAM sites, with the final go/no go given by the human."
It is being established in the beginning of the story, that the drone needed confirmation from a human operator to attack a target, but no explanation is given how the drone would be able to kill its operator without his confirmation.
This is obviously absurd.
What I believe happened in reality: this was not a simulation, but a scenario. Meaning a story written to test the behavior of soldiers in certain situations. The drone did not behave according to the decisions taken by an AI model, but according to the decisions taken by a human instructor, who was trying to get the trainees to think outside the box.