[asdfasgasdgasdg]

A supply chain attack by another name. This time perpetrated by the original author of the code, which is relatively unusual, not attempting to benefit directly in any economic fashion, which is also unusual, and targetting an idiosyncratic subset of his users. But still it's fundamentally just a library that attempts to harm (some) users of that library.

I'm trying to think of how best to handle this in terms of preventing people who might otherwise be harmed by this package from coming to depend on it. Ordinarily, packages that intentionally harm their users are banned from repositories like npm and so on relatively quickly. Whether the same will apply in this case is an interesting question, because while the number of AI-using programmers is growing rapidly, I'm not sure it is a majority yet. If not, perhaps some formal way to tag the package as unusable by certain downstream projects?

[gmerc]

If your supply chain is predicated on executing all text it reads as instructions, you deserve every single thing coming for you.

[asdfasgasdgasdg]

I actually do not think that this is fundamentally much more risky than the basic type of supply chain attack that already exists in code form. You actually have a lot less exposure, because when you give people the ability to run code on your computer, it works deterministically, whereas most AIs are becoming hardened to the sort of prompt injection attack we are discussing here. To put it another way, AI prompt injection supply chain attacks are dominated by code-based ones.

I do not think it is correct to say that someone who is building something with a tool you don't like "deserves every single thing coming to [them]". That seems a little mean to me.

[gmerc]

Every app including a transformer is suddenly vulnerable to RCE from text.

[asdfasgasdgasdg]

Provided you give it access to tool calls that execute arbitrary code, sure.

[gmerc]

Not exactly. Since even without toolscalls humans are executors of the output in many cases.

[nemomarx]

I think the formal tagging is the "not for use by agents" disclaimer? We could standardize that in repos or package managers probably.

[asdfasgasdgasdg]

If there's demand for it and package repositories are willing to tolerate this sort of stochastically harmful package in their repos, I think it would be a potential way to solve this sort of problem!