[raincole]

> as an assistant that helps me make decisions, and ensures that I'm in the driver's seat

This... quite describes what the current AI feels like to me. So I really don't see a problem. If AI does too much for you, you should, uh, just prompt it to do less.

Everyone talks about how their AI escapes sandbox and virtualization or delete their entire repos or something. Perhaps they're using a skynet I'm not aware of. From my experience when I added "discuss; don't edit any file yet" at the end of the prompt I've never seen the LLM go against my will and edit the files anyway.

[hollowturtle]

> From my experience when I added "discuss; don't edit any file yet" at the end of the prompt I've never seen the LLM go against my will and edit the files anyway.

depends how long conversation gets and he will still ignore your instructions randomically. I think shorter sessions are better, I do have a similar prompt saved on the memory of the agent and way more times that I can tolerate, he candidally ignores it

[samrus]

Shorter sessions are most definitely the only way to go. I ensure thay by having a clean context handoff process and then using one session for one task and having it dump context when its done or gets too big (~400k tokens)

[existencebox]

I expect there's a probabilistic factor to it, among other things.

I use the almost the same addendum as you ("do not make, commit, or push any changes without my approval") and it broadly seems to work in my day-to-day dev. However, I also set up some agents that trigger on incoming requests from my dev teams to automate things that used to be rote/manual tasks. _That_ is where I see the issues, I suspect simply because it has so many more opportunities to go off script.

The most common failure mode is that it will, despite all directives to not make changes without approval, start writing unit tests. (I do find this equal parts head-shaking and amusing; At least it chooses something relatively 'benevolent' vs. the aforementioned skynet.)

[Sharlin]

I haven’t used these tools that permit LLMs to make changes on their own, but I can’t help being baffled if there’s no big red switch that changes the system prompt and disables any and all write access to the outside world.

[SyneRyder]

There is - in the Claude Code harness there's a dedicated Plan Mode that doesn't edit any files, or a mode where you can manually approve each edit.

If you're running Claude via -p automated mode, you can set all kinds of permissions and access to each individual editing tool, much like you would with firewall software.

It doesn't edit the system prompt, it blocks access to the tools itself. Just make sure you haven't given the AI the ability to edit its own configuration files, or you'll potentially have an OpenAI v Hugging Face situation.

[blks]

I think the best way is to run agents under Unix user that has very limited rights (eg only read only) for project folder and the rest of the system. It’s kind terrifying how people run agents on their computers without any hard limitations, just hoping they will follow instructions and not do anything malicious. This is basically a trojan waiting to happen, either trying to solve something by all means possible, or getting prompt injection

[nemomarx]

I do wonder if "without my approval" is looser than "discuss only, do not make edits"? I'm not sure how much that finegrain thing matters in prompts nowadays tho

[jameshart]

One problem is that agents are notoriously bad at handling ‘me’ vs ‘you’ over an extended conversation. ‘Without my approval’ far back in the context of an extended session gets fuzzy about who ‘me’ is.

[cwnyth]

This has always seemed strange to me. It will offer a suggestion, I'll make a comment about a suggestion, and it returns with, "Your suggestion..." The further distance from the suggestion, the more likely it confuses who made it.

[Izkata]

Possibly just a general issue, rather than a long conversation one: https://news.ycombinator.com/item?id=48799439

[bitwize]

This is because of the nature of what it's doing. It's still just predicting tokens behind the scenes; it's just treating the entire conversation between you and it as one long chunk of text for which it must predict the text to come. But the model has no way of keeping track of who said what, and it's not really conversing with you the way you are with it. So screwups like confusing "my" for "your" are to be expected.

Much of the transformative power of LLMs has come from dumb logic wrapped around them that either changes what is fed to them ("memories", "thinking/reasoning", etc.) or how their output is interpreted/acted upon (agent harnesses, loops, etc.). For this reason, Anthropic could release "Claude Necronomicon" tomorrow and tell us it could hack the planet and gain control of the world's nukes, dooming us all, but until it's actually hooked up to something that enables the execution of those commands, it can't do squat. Of course, that's no reason not to worry since some idiot is likely to build that torment nexus (and they may well work for Anthropic), but at the end of the day the thing is still just predicting tokens and everything else it seems to be doing is being faked with clever massaging of the input or output.

[manwe150]

That’s not really the issue. The issue I have (sometimes) is that it might discover it has read-only access to most places (.git, github) and so sometimes when it thinks it “finished” it will discover those are read only and try to find workarounds (erase the read-only files, hand edit the .git folder, make new forks to push to those instead)—none of which a human would normally consider reasonable necessary effort. Most of those attempts do fail, but it turns out that sometimes just being maximally helpful in following orders strictly can be quite surprisingly adversarial, entirely by accident