[pjm331]

Yeah i have a similar setup at the harness level - a “devlog” at the end of every session about the experience developing, what could have been better, what was confusing

And also files issues for blockers

I’ve absolutely caught things and made improvements just from skimming them occasionally - they are particularly useful when you get a PR that makes you scratch your head

But I’m definitely not taking full advantage of all the feedback coming in yet

I have to imagine parsing signal from noise there is a massive challenge when it’s other agents that are using your MCP and not just your own

[what]

Why are you asking an LLM what could have been better or was confusing? It literally has no idea.

[rsfern]

True. But it has no idea that it has no idea, so it might be able to look back at the session trace and pattern match its way to actionable feedback?

[pjm331]

Listen friend I’m also in the camp of LLMs are just math and data and don’t actually “have an idea” about anything but that doesn’t make them any less accurate/effective at generating tokens

I’m not sure why this prompt is different than any other in that regard

[simonw]

If they try something and it doesn't work and they then try something else and it DOES work that's concrete feedback.

[what]

Are you saying the machine can tell you what confused them?

[simonw]

Yes. Try a prompt like this:

> In our recent conversations were there any instances where you attempted a tool call, got an error, then adjusted your tool call and were successful? Suggest improvements to the tool instructions that would have avoided the initial error.

A year ago I don't think this would have worked. The frontier models have been trained to be able to write prompts for themselves, mainly so they can effectively use sub-agents.

[datadrivenangel]

they're surprising good at this. The next tokens after a confusing string of tokens when asked to explain the confusing parts are pretty accurate!

[catlifeonmars]

You won’t get grounded results by asking for feedback. The LLM doesn’t know its own internal state so while you may get a plausible and contextually significant answer, it isn’t going to be an accurate one.

[kolinko]

Are you talking from experience or from theory?

I have a reflections mechanism, which launches right after the session ends (when it’s still in cache, so it’s almost free) and it catches a ton of problems with tool/skill/system design, all the time. Low level stuff can be fixed almost automatically (e.g. agent funbled 10 turns bc tool’s docs were wrong), high level too but human gating is useful.

If you try doing it any other way you have to pay for input tokens, not cache read, so it’s 10x more expensive.

[catlifeonmars]

Theory. I guess my question is: why do you think it works so well. What’s the mechanism behind it?

It feels like it would be relatively random across the range of plausible results, but it sounds like you are finding otherwise.

[kolinko]

Why would it be random? Perhaps we’re talking about two different things?

I’m talking about asking an agent at the end of the session how well the session went - what tools it found useful, what it found confusing and whatnot. Agents do gladly complain about doing pointless work or fighting with the harness :)

[catlifeonmars]

Let me put it this way. How does the model know what it finds confusing?

I’m sure it works great fwiw, also the use of a hot cache is clever.

[kolinko]

I have a similar system and it does wonders. One of the questions is also to ask for just freestyle opinion of the task and the replies are hilarious at times.

“I existed for one word. The entire point of my life was that my process boot side-effected a module reload. 'ok'."

“ "The overhead-to-work ratio was astronomical."

“the koleo purchase turned into a boss fight with an undismissable Vue modal and a WKWebView that kept eating its own cookies."