[kolinko]

Are you talking from experience or from theory?

I have a reflections mechanism, which launches right after the session ends (when it’s still in cache, so it’s almost free) and it catches a ton of problems with tool/skill/system design, all the time. Low level stuff can be fixed almost automatically (e.g. agent funbled 10 turns bc tool’s docs were wrong), high level too but human gating is useful.

If you try doing it any other way you have to pay for input tokens, not cache read, so it’s 10x more expensive.

[catlifeonmars]

Theory. I guess my question is: why do you think it works so well. What’s the mechanism behind it?

It feels like it would be relatively random across the range of plausible results, but it sounds like you are finding otherwise.

[kolinko]

Why would it be random? Perhaps we’re talking about two different things?

I’m talking about asking an agent at the end of the session how well the session went - what tools it found useful, what it found confusing and whatnot. Agents do gladly complain about doing pointless work or fighting with the harness :)

[catlifeonmars]

Let me put it this way. How does the model know what it finds confusing?

I’m sure it works great fwiw, also the use of a hot cache is clever.