[existencebox]

I expect there's a probabilistic factor to it, among other things.

I use the almost the same addendum as you ("do not make, commit, or push any changes without my approval") and it broadly seems to work in my day-to-day dev. However, I also set up some agents that trigger on incoming requests from my dev teams to automate things that used to be rote/manual tasks. _That_ is where I see the issues, I suspect simply because it has so many more opportunities to go off script.

The most common failure mode is that it will, despite all directives to not make changes without approval, start writing unit tests. (I do find this equal parts head-shaking and amusing; At least it chooses something relatively 'benevolent' vs. the aforementioned skynet.)

[Sharlin]

I haven’t used these tools that permit LLMs to make changes on their own, but I can’t help being baffled if there’s no big red switch that changes the system prompt and disables any and all write access to the outside world.

[SyneRyder]

There is - in the Claude Code harness there's a dedicated Plan Mode that doesn't edit any files, or a mode where you can manually approve each edit.

If you're running Claude via -p automated mode, you can set all kinds of permissions and access to each individual editing tool, much like you would with firewall software.

It doesn't edit the system prompt, it blocks access to the tools itself. Just make sure you haven't given the AI the ability to edit its own configuration files, or you'll potentially have an OpenAI v Hugging Face situation.

[blks]

I think the best way is to run agents under Unix user that has very limited rights (eg only read only) for project folder and the rest of the system. It’s kind terrifying how people run agents on their computers without any hard limitations, just hoping they will follow instructions and not do anything malicious. This is basically a trojan waiting to happen, either trying to solve something by all means possible, or getting prompt injection

[nemomarx]

I do wonder if "without my approval" is looser than "discuss only, do not make edits"? I'm not sure how much that finegrain thing matters in prompts nowadays tho

[jameshart]

One problem is that agents are notoriously bad at handling ‘me’ vs ‘you’ over an extended conversation. ‘Without my approval’ far back in the context of an extended session gets fuzzy about who ‘me’ is.

[cwnyth]

This has always seemed strange to me. It will offer a suggestion, I'll make a comment about a suggestion, and it returns with, "Your suggestion..." The further distance from the suggestion, the more likely it confuses who made it.

[Izkata]

Possibly just a general issue, rather than a long conversation one: https://news.ycombinator.com/item?id=48799439

[bitwize]

This is because of the nature of what it's doing. It's still just predicting tokens behind the scenes; it's just treating the entire conversation between you and it as one long chunk of text for which it must predict the text to come. But the model has no way of keeping track of who said what, and it's not really conversing with you the way you are with it. So screwups like confusing "my" for "your" are to be expected.

Much of the transformative power of LLMs has come from dumb logic wrapped around them that either changes what is fed to them ("memories", "thinking/reasoning", etc.) or how their output is interpreted/acted upon (agent harnesses, loops, etc.). For this reason, Anthropic could release "Claude Necronomicon" tomorrow and tell us it could hack the planet and gain control of the world's nukes, dooming us all, but until it's actually hooked up to something that enables the execution of those commands, it can't do squat. Of course, that's no reason not to worry since some idiot is likely to build that torment nexus (and they may well work for Anthropic), but at the end of the day the thing is still just predicting tokens and everything else it seems to be doing is being faked with clever massaging of the input or output.