[simonw]

I heard a neat tip recently about API design for agents: give them a way to send you feedback.

The example I heard was an MCP with a "feedback" tool which had a tool description saying that coding agents should call that any time they had trouble figuring out how to use the rest of the MCP.

I really like this. It's super cheap to implement and I expect you'd get a bunch of actionable signal in amongst the noise.

[hdjrudni]

My work recently changed an agent API. It was a certain CLI tool built for agents which I had copied into one of my Skills.

What they did was kind of clever. When the agent calls the old API/uses old args, I guess they told it to tell the human to fix their skill and 'upgrade' to the new API. They even provided a skill upgrader skill.

Anyway, my agent tried calling it the old way, must have received some sort of error message and then diligently reported to me that my skill needed upgrading and we should be doing it some new way.

Kind of like reverse feedback I guess? Not the AI giving feedback to them, but them giving feedback to me via my AI.

[kolinko]

Good one! Also, a simple thing - when your service gets miscalled (wrong params either via cli or http) instantly show help/explaination of the proper param uses. Can save a few turns of agents trying to figure out what’s wrong. Helps humans as well :)

[pjm331]

Yeah i have a similar setup at the harness level - a “devlog” at the end of every session about the experience developing, what could have been better, what was confusing

And also files issues for blockers

I’ve absolutely caught things and made improvements just from skimming them occasionally - they are particularly useful when you get a PR that makes you scratch your head

But I’m definitely not taking full advantage of all the feedback coming in yet

I have to imagine parsing signal from noise there is a massive challenge when it’s other agents that are using your MCP and not just your own

[what]

Why are you asking an LLM what could have been better or was confusing? It literally has no idea.

[rsfern]

True. But it has no idea that it has no idea, so it might be able to look back at the session trace and pattern match its way to actionable feedback?

[pjm331]

Listen friend I’m also in the camp of LLMs are just math and data and don’t actually “have an idea” about anything but that doesn’t make them any less accurate/effective at generating tokens

I’m not sure why this prompt is different than any other in that regard

[simonw]

If they try something and it doesn't work and they then try something else and it DOES work that's concrete feedback.

[what]

Are you saying the machine can tell you what confused them?

[simonw]

Yes. Try a prompt like this:

> In our recent conversations were there any instances where you attempted a tool call, got an error, then adjusted your tool call and were successful? Suggest improvements to the tool instructions that would have avoided the initial error.

A year ago I don't think this would have worked. The frontier models have been trained to be able to write prompts for themselves, mainly so they can effectively use sub-agents.

[datadrivenangel]

they're surprising good at this. The next tokens after a confusing string of tokens when asked to explain the confusing parts are pretty accurate!

[catlifeonmars]

You won’t get grounded results by asking for feedback. The LLM doesn’t know its own internal state so while you may get a plausible and contextually significant answer, it isn’t going to be an accurate one.

[kolinko]

Are you talking from experience or from theory?

I have a reflections mechanism, which launches right after the session ends (when it’s still in cache, so it’s almost free) and it catches a ton of problems with tool/skill/system design, all the time. Low level stuff can be fixed almost automatically (e.g. agent funbled 10 turns bc tool’s docs were wrong), high level too but human gating is useful.

If you try doing it any other way you have to pay for input tokens, not cache read, so it’s 10x more expensive.

[catlifeonmars]

Theory. I guess my question is: why do you think it works so well. What’s the mechanism behind it?

It feels like it would be relatively random across the range of plausible results, but it sounds like you are finding otherwise.

[kolinko]

Why would it be random? Perhaps we’re talking about two different things?

I’m talking about asking an agent at the end of the session how well the session went - what tools it found useful, what it found confusing and whatnot. Agents do gladly complain about doing pointless work or fighting with the harness :)

[catlifeonmars]

Let me put it this way. How does the model know what it finds confusing?

I’m sure it works great fwiw, also the use of a hot cache is clever.

[kolinko]

I have a similar system and it does wonders. One of the questions is also to ask for just freestyle opinion of the task and the replies are hilarious at times.

“I existed for one word. The entire point of my life was that my process boot side-effected a module reload. 'ok'."

“ "The overhead-to-work ratio was astronomical."

“the koleo purchase turned into a boss fight with an undismissable Vue modal and a WKWebView that kept eating its own cookies."

[benswerd]

One strategy I like for this is a notes field on every single MCP route. Lets the AI optionally just decide to tell you things.

[tekacs]

I love this, although I can't help but think that a lot of agents will - for better or for worse - send you a bunch of PII.

[ktanishqk]

a feedback tool idea is good (mintlify landed on smth similar) but it's just the starting point. what we found is feedback < support. agents love resolution because anything less is still a blocker in their workflow. and it can't be MCP specific because your curl and CLI users need the same surface.