[jaggederest]

That's interesting because I would feed that benchmark back into the agent and loop over it, to see how much faster you could get it, and agents are really good at that kind of recursive optimization. And I would definitely add at least a simulated 1GiB write test, probably a real one honestly, if I was building something like that.

At least with agent-run tests I care about loop speed a lot, but I care about complete coverage more, so having the odd heavy weight full stack integration test is fine, I think.

[dprkh]

You're right. This was just a performance issue, but what if next time it is a corruption bug or a security vulnerability or really anything that can cause real consequences if happened in production? I don't think that LLM systems are inherently bound to have this flaw, but I think that we are pretty far from harnesses and algorithms becoming advanced enough so that the LLM system can kind of continuously evaluate its output and ensure it is good in all aspects.

[jaggederest]

I don't know about that, Fable is, when properly guided, a better engineer for those things than I am. Narrow breadth, weird priorities, myopic and ivory tower as hell, but superhuman. Maybe that says more about me, or maybe not, but certainly it's caught bugs I would not have, and point it at things like a fuzzer, woo buddy, it has been many years since I broke out valgrind and nailed down a memory leak, but it sure can.