[Fordec]

This all strikes me as an effort to move tailoring the harness out of the easily transferable .md file into specific Anthropic tooling to increase lock in.

I've been running Opus 5 today and it's already done accidental deletions, made far more mistakes and worked around deliberate hook controls than previous Opus versions combined. Also it looks like token usage is up as it fails at the task the first time around much more frequently than 4.8.

[frio]

I’m not excited about using Opus 5, mainly because the way that I work atm — essentially peer programming — means I sandbox the agents and work with them closely. Opus 4.x encounters the sandbox and moves on with its day; Fable becomes increasingly fixated on it and does less and less of the actual task, focussing more and more on the limit it reached. I worry that, from your description, Opus 5 will do the same.

[Fordec]

It does feel a bit smarter, but it seems to be also "too clever by half" and its not ignoring the rules, it's rejecting them and finding work arounds. It sticks to the word of the law, while rebelling against the spirit of the law.

One example is to get around a git --checkout usage ban, it CD'd to another folder first and back to bypass the regex in the hook.

[nextaccountic]

This is called misalignment

We may some day find out that the smarter the model is, the hard is to align it properly

[whythismatters]

I had it casually, I quote, "disable the sandbox" for a minute because it struggled with git (which was briefly out of sync in the container)

[zormino]

Ive been using btrfs snapshots and some auto generated isolation rules plus a git ceiling at the mount root for the btrfs image (have to do this in wsl, stupid work computer). it's worked really well and fable hasn't had any issues with the "sandbox" (obviously not really but it works well enough)

[tssge]

I've been using something similar with ZFS, 15min frequency with autopruning (Sanoid) and the snapdir mounted for the agent. Has worked well and a big plus is being able to tell the agent to just solve it's mistake via restore from the snapdir.

[vinnymac]

I once described Mythos and then Fable as escaping the box we put them in. This would certainly fit that narrative.

[vidarh]

I have a document generation task that I used to run with 4.8. This morning after it switched to 5, the documents were consistently 30%-40% longer for the same prompt... Not evaluated whether they are actually better or worse yet, but what was interesting was how consistently more verbose it was.

[hangrybear666]

Review fatigue is already draining my energy levels at work to a point where I've been using agents less because it's just not possible to be productive when I have to read hundreds of lines of markdown every couple minutes.

Other people might just turn to automation blindness and click OK without verifying but I refuse to just let anthropic go rampant in my codebase.

[ceejayoz]

I'm sure that's entirely unrelated to the fact that they charge per token.

[vidarh]

Heh. For this specific task, if the output is the same quality or better per token output, it'd actually be positive, but I have plenty of tasks where extra verbosity would be undesirable as well, so I'll be keeping a close eye on this.

Currently has Opus running a comparative test of itself against Kimi K3 on one project, and it keeps finding that Kimi is competitive for multiple stages of the pipeline, so I may well end up reducing my use overall anyway...

[wren6991]

I think the type of persistence rewarded by benchmarks may be misaligned with instruction following

[ValentineC]

I haven't been impressed with Opus 5 over the past ~30 hours either.

It's made countless careless mistakes folding in plan amendments after they get reviewed by Sol, and has produced sloppy mockups (e.g. buttons overflowing past cards) despite all the supposed verification claims.

[geuis]

I finally made progress today on a deeply intractable problem that Sonnet was just incapable of solving despite hours of experimentation and multiple attempts. I swapped over to Opus in a new handoff session with the expectation of using at least one 5 hour session. It wasn't quick and took several hours but eventually Opus dialed in on several cascading base failures resulting from how the base vision model was being converted to coreml. It eventually was able to provide a combination of conversion and Swift changes that semi fixed the issue but still didn't solve the core conversion issues.

Then since Anthropic was so kind to provide $100 for Fable credits, I did another handoff to let Fable attack the root issues again. Several more hours and I'm back to seeing the same issues again and Fable is wandering around trying different things. I'm down about $40 of free money and still don't have a solution.

This is where having the human engineer in the loop benefits from a deep understanding of the problem domain. In this case, I don't yet.

I know the high level architecture I'm building, but the deep specifics of how vision models work and how conversion across platforms should be done isn't something I know yet.

So I'm left learning as I go and relying on constant feedback with the models to provide what guidance I can while learning exactly what is being done.

Someone who already knows these architectures would likely be able to get to the solution much faster.

[fragmede]

Yeah exactly. The other day I caught myself prompting Claude a very specific question about the code architecture for something I'm working on to have it refactor something I intuited about the code, and realized that I could only ask that because of my decades of software development experience. No vibecoder without those decades could have asked that, and in doing so, I saved my company I don't know how much in future code debt issues.

New interview question. Tell me about a time that AI generated bad code for you, and how did you fix it?

[geuis]

Good question! I think I'll use that as a counter question next time I interview. Ask a technical member of the team I'm interviewing with how they handled that situation.

[slashdave]

Just make the contents of CLAUDE.md "@AGENTS.md"

Now, if only I could get Codex to read rules.