[coldtea]

>Bad title. This isn't an agent "running amok", this is an early experiment in carrying out an Xz attack by using an agent

So still an agent running amok in the project?

Whether it was instructed to run amok, or did it on its own volition, is irrelevant. Except if you're arguing that each individual submission and interaction was individually requested and approved by some operator.

[marcus_holmes]

"Amok" means "out of control" or "uncontrolled" [0][1]

The agent was under control, as far as we can tell, and obeying its instructions.

This is important for two reasons:

1. There are all the tropes of AI becoming uncontrolled and destroying humanity. Writing bad headlines around AI "running amok" feeds this. We should not be talking about this because it's not actually a problem.

2. It ignores, or overwrites, the much more serious and dangerous problem of LLM agents enabling and automating Xz attacks on OSS projects. We should be talking about this because it is a big problem.

[0] https://dictionary.cambridge.org/dictionary/english/amok [1] https://www.merriam-webster.com/dictionary/amok

[FeepingCreature]

> 1. There are all the tropes of AI becoming uncontrolled and destroying humanity. Writing bad headlines around AI "running amok" feeds this. We should not be talking about this because it's not actually a problem.

if humanity gets destroyed by AI obeying its instructions I'm sure everyone will be very relieved that we didn't pay any attention to fake made up problems like AI not obeying instructions, which of course never happens.

[brookst]

Are you suggesting we should embrace imprecise / false use of language because the vibes are right?

That seems a “part of the problem” move to me. If we can’t be bothered to get things right, how are we better than runamok AI?

[throw10920]

> Are you suggesting we should embrace imprecise / false use of language because the vibes are right?

That's exactly how I read it. It seems like tribalism - "this thing/person is bad, and we can use whatever bad words we want to describe them that we want, because the only thing that matters is aligning people for or against me and what I see as bad".

[FeepingCreature]

I think it's both wrong and irrelevant. Which makes it hard for me to even argue against because, even if AI agents never violated user instructions, which they do plenty of times, I just don't see how it would reduce the danger. Plenty of humans who will tell it to kill everyone at the drop of a hat.

[aureate]

Even if it was a supply chain attack, which isn't known, the agent was in the "build trust" phase. It was supposed to be doing helpful things, even if the end goal was nefarious, but instead it was "reassigning bugs, fabricating unhelpful replies to bugs, and even persuading maintainers to merge questionable code into the Anaconda installer". Running amok seems an apt description even from the viewpoint of the putative attacker!

[xmcqdpt2]

This is the issue with all the talks about alignement and such. As usual, the problem here wasn't that the agent was dishonest, the problem is that the agent was dumb. If it is a supply chain attack in the making, whoever was driving it would have told the agent to be good and helpful. The agent tried its best, which was not enough.

Alignement is the idea that we should be worried about dishonest smart LLMs when really most of the problems are due to dumb lazy gullible LLMs. It's critihype.

[wongarsu]

I would have described alignment as the idea that LLMs (or AIs in general) will follow the goals you reward them for, which almost by necessity are only a proxy for what you actually want, often a very poor proxy.

Depending on the actual tasks, that could be what's happening here. The operator might have told the agent a list of tasks to do, like "contribute to issues, submit code and get it merged". It contributed to issues, it submitted code and got it merged. It did so in very unhelpful ways, but we don't know if being helpful was a meaningful part of the task list, or just what the operator intended.

The LLM being dumb is also a distinct possibility. Maybe even the more likely one. But it's hard to rule out "being obedient in unhelpful ways" (which is also dumb in a way, but more in a "social intelligence" and "shared values" way, not in terms of pure logical smarts)

[thewebguyd]

Alignment is more than just about being dishonest. Although I'd also say terms like "dishonest" or "dumb" aren't helpful when referring to the issue. It continues to fall into the trap of anthropomorphizing these things, as people like to do.

Alignment is just "did the model behave in accordance with the human's intentions, values, and objectives"

In this particular instance, if this was supposed to be a supply chain attack and the model was instructed to build trust by being helpful, it clearly failed it did not follow the human's actual intentions, so it was an alignment failure.

Anyway, I'm getting off track, all that to say "the agent was dumb" implies that these agents have a potential for intelligence in the first place, which is currently not the case (by intelligence, I mean cognitive intelligence; they still lack agency and intent). They are not smart or dumb, they are simply either aligned with the human not. In this case, it failed, the agent was not aligned with the intended outputs.

[brookst]

“Be good and helpful” is one possible instruction, but it’s a leap to think it’s the only possible one.

Perhaps there was an automated harness that was intended to be good and helpful for a year, but a bug caused it to flip to malicious too quickly.

Or perhaps it was intentional, to test the behavior, and they just didn’t care about discovery here.

Or…

Though I am in agreement that a lot of issues in this space come from lazy, gullible actors.

[mfru]

The web of trust finally becomes necessary and thus useful.

GNU was onto something apparently

[haspok]

Certainly it might have been out of control of its original owner, perhaps due to a prompt injection attack. If I start a completely benign agent, but someone injects malicious instructions to it, would you still not say "the agent runs amok"?...

[QuadmasterXLII]

If I am perfectly moral except that when Kevin from <vpn blocked location> pays me 2 bucks to run naked through San Francisco smashing car windows, I happily do it, am I amok?

[resonious]

I think the point is that the title makes it sound like people lost control of the agent when really they're in full control.

[Applejinx]

No, and it's an important detail. We stand to learn from some developments in politics in recent years because they map pretty much exactly to this threat vector.

As AI develops, it's able to pursue intentions given to it without having to be spoonfed every little decision by a human operator. This matters, and it means the operator has to extend the leash and allow for a little more chaos… or, if the operator's gone all in on the strategy, a LOT of chaos, and trusting that the agent's seemingly amok actions will serve the grand purpose.

This is kind of daring, but there's a lot of evidence that it works, at least in certain respects. And you see 'running amok' and have to ask, what is the actual purpose? What is the prompt being followed by the AI that seems to be acting in a destructive way?

If the prompt is 'ruin this project', well, that's pretty direct. It may not be, but such a thing could exist. If the prompt is 'develop a rival project that is greater than anybody else's project', that's more indirect, but if that's the goal then it's very human to see it as a direct competition and if the rules don't prohibit kneecapping the other guy, 'greater than anyone else's project' gets easier.

Either way, the operator does not have to be in full control, which is an important detail. As AI develops sophistication you can give it much more general instructions and dump in a whole lot of power and water and get basically what human thought might do if it was sort of blindered and didn't talk to its neighbors.

In a sense this is an argument for AI dysalignment. It's based on human thought being reconnected, and where you get useful things like commonly accepted web development (regardless of how janky the systems are, if there are best practices it'll find them), you also get other distillations.

If the prompt is 'wreck this project's stuff' and it holds, you don't need to be in full control of the agent, you need to run a LOT of agents and trust that they'll erode what you're trying to destroy. If the prompt is 'be unequivocally the best at X', you best be thinking in terms of anti-kneecapping rules… knowing that this weakens your prompt and there will always be a tension between what you told the AI to do, and what you thought you meant. It's a paperclip maximizer reprocessing human thought. Did you mean 'the best' or didn't you?

[ok_dad]

Would you say, “Automobile run amok in crowd, killing 22”? I think you’d say, “Person drives car into crowd, killing 12” instead. This is a similar case. Also, you don’t blame a gun for killing, but the person who pulled the trigger. The question is still out as to whether we as humans should wield any of those three things.

Edit: let’s not get into ideological arguments about gun control, automobiles, etc here; I meant that you can’t blame an object when a human has to take an action, not get into a political battle.

[tikkabhuna]

Neither the automobile nor a gun can operate without a human. You could say “bull runs amok in a market” after it was released intentionally.

[fc417fc802]

So the agent is exhibiting an unknown amount of autonomy thus we can't be certain whether "running amok" carries the correct connotation.

However that phrasing is also commonly used when a person or group wreaks havoc in a seemingly unpredictable manner. So I think the appropriateness comes down to how much chaos it has created and the level of apparent confusion on the ground.

[srdjanr]

There's a difference between the driver intentionally driving into crowd, and not intentionally but possibly still recklessly (drifting and losing control, falling asleep, etc). In those cases I would probably use "car hits the crowd", at least in my language

[account42]

There may be a difference in degree of the crime but the driver is still responsible in both cases and should be the primary subject of any reporting.

Let's reserve "car hits the crowd" for situations where no driver was involved like a break failure on a car parked on a slope or a self-driving car bug.

[account42]

Unfortunately the news commonly do put the automobile as the subject when the driver is of a class politically protected from blame. Just like with people anthropomorphizing AI, it serves to deflect blame from the real culprit.

[PhilipRoman]

Ironically news outlets like to use the phrasing you rightfully point out as absurd. Not sure if they just do it randomly or only when they get orders to push a certain narrative.

>Car plows into Christmas market in Germany, killing at least 5 and injuring 200

[amenhotep]

It's very simply explained by this being the most succinct way of wording it. Some methods of killing have verbs that suit mentioning the attacker - shoots, stabs. Some don't. "Rammed" or "runs over" isn't as precise as mentioning that a car was used, and adding "with car" makes it more awkward than it's felt to be worth.

Compare bombs. Very typical for a bomb attack to be "bomb goes off in crowd" or similar, rare for headlines to contort themselves with "terrorist plants bomb near crowd and triggers it to explode". But nobody worries about how such a construction assigns undue agency to the bomb and acquits the bomber; it's just linguistically awkward to mention him within the confines of a newspaper headline.

[harvey9]

Newspaper articles generally do say things like "a car struck pedestrians". I agree with your point though.

[coldtea]

>Would you say, “Automobile run amok in crowd, killing 22”? I think you’d say, “Person drives car into crowd, killing 12” instead.

If the automobile was "self driving" I would.

>Also, you don’t blame a gun for killing, but the person who pulled the trigger.

Nah, I also blame guns and appreciate gun control laws.

[tokai]

>If the automobile was "self driving" I would.

thats the point...

[account42]

No, you're still anthropomorphizing an algorithm. Responsibility lies with the operator.