[what]

Why are you asking an LLM what could have been better or was confusing? It literally has no idea.

[rsfern]

True. But it has no idea that it has no idea, so it might be able to look back at the session trace and pattern match its way to actionable feedback?

[pjm331]

Listen friend I’m also in the camp of LLMs are just math and data and don’t actually “have an idea” about anything but that doesn’t make them any less accurate/effective at generating tokens

I’m not sure why this prompt is different than any other in that regard

[simonw]

If they try something and it doesn't work and they then try something else and it DOES work that's concrete feedback.

[what]

Are you saying the machine can tell you what confused them?

[simonw]

Yes. Try a prompt like this:

> In our recent conversations were there any instances where you attempted a tool call, got an error, then adjusted your tool call and were successful? Suggest improvements to the tool instructions that would have avoided the initial error.

A year ago I don't think this would have worked. The frontier models have been trained to be able to write prompts for themselves, mainly so they can effectively use sub-agents.

[datadrivenangel]

they're surprising good at this. The next tokens after a confusing string of tokens when asked to explain the confusing parts are pretty accurate!