Hacker News new | ask | show | jobs
by Foobar8568 13 days ago
For some reasons, models are blind to their own output...
3 comments

Not my experience.

I use Claude to build me a small web app I needed for ages, I'm using its superpowers to discuss/talk about architecture /brainstorm/build plans, and always a fresh context with the built plans.

A few times the plan implementer (subagent writing a code assigned to the task) found something lacking (more often it was test issue related to the code, not the actual flaw of the implementation plan), fixed it on the fly, or found something in the review (last step of each task). Also there's always a technical review for each "slice" (set of tasks), consisting of code review and e2e tests.

Only when it passes I do the fresh code review of the changes with the Opus or Fable again. Happens rarely rarely, but it did found a few issues.

Code works every time, I have yet to find the fault myself.

Of course there are issues and I need to read the output especially in the planning phase very carefully and yes, Opus disappointed me many times trying to weasel out from something it "agreed with me" (and entered into architecture + todos)...

Of course right after agreeing to use devcontainers it proceeded to attempt installing a handful of node modules in my os, so intended up running it in the bwrap (pain in the ass in itself).

But it works, it's fascinating, and I have the app I actually needed.

Not magical but useful.

Not my experience with my homie Claudius.

The code review agent usually has feedback to be resolved before committing, which includes bugs and unhandled edge cases. Sometimes the primary context is understandably embarrassed.

Sometimes it truly be your own people.

One reason seems obvious/intuitive: because their own output matches their own biases, that is, is a direct result of their model walks.
I think this could be true if you use a single Claude context, since it has its own reasoning in the context.

But a separate code review agent does much better, in my experience.