[o10449366]

/goal has replaced plan mode for me. This is the pattern I use for 95% of my AI work now:

1. Read X feature of Y and tell me when you fully understand it (if there's any detail missing in the summary, repeat until the context is primed)

2. What time is it?

3. /goal Spend X minutes from $time writing a technical design doc on $feature. There must not be any vague language or ambiguity in the document. Read carry_forward_requirements.md and testing_best_practices.md and explicitly incorporate them into the document you write. The document should be executable for a contextless implementer when done and include specific code and document references and changes needed. Spend the full X minutes working on and reviewing this document - do not quit early and wait

Even just spending 10 minutes forcing GPT to write a design doc results in much more robust plans than plan mode, in my experience, and saves time I would spend iterating on the initial plan mode draft anyway.

[embedding-shape]

> 3. /goal Spend X minutes from $time writing a technical design doc on $feature.

Hmm, I feel like this is akin to making a recursive function have a exit condition not based on what it actually did/found, but based on how long time it took.

I'm always using /goal with explicit goals that the agent needs to achieve. Time-bounding them wouldn't make sense, I want something specific done regardless of how long time it takes.

So instead I'd put goals on what the design/architecture needs to achieve, and for the model to continuously check the outcome against these, then finish when everything is achieved. Doesn't really matter if it takes 10 minutes or 10 hours, which for me is a bit the point of /goal in the first place, otherwise I'd just use the agent normally.

[lonelyasacloud]

Leaving to run unhindered makes sense when the agents understand both the complete domain context and can be supplied with a strong exit criteria that they understand well.

Otherwise time boxing is both going to help stop entities from wandering off into the weeds. And also communicate expectations from the commissioner about the expected effort levels and output quality requirements.

Empirically in human world, get very different results when an employee, particularly a junior, is asked to spend 1/2 a day on a work package, a week or are left completely to their own devices.

[o10449366]

I think like the other commentator noted, I don't do this blindly, as I also wouldn't if I was instructing a junior engineer. I have a rough sense of how long it would take me to research all angles of the task. 5 minutes for a complicated feature is too short for a comprehensive plan. Letting it run for a few hours is also probably unproductive. But 10-60 minutes depending on complexity with followup review is a nice sweet spot. Just my $0.02.

[drivebyhooting]

The proof prompts do say to spend at least 8 hours.

[illliillll]

Well, I’ve been having 5.6 sol work on tasks like “find every OTA app on the internet”

I find explicit time bounds are useful for tasks like this, otherwise the LLM will almost certainly return too early.

[bob1029]

The hypothesis generation phase is looking like the most critical part of having an agent reliably hit your targets.

Simply starting in the correct part of the search space is probably the biggest predictor of success. Forcing one big loop to fight its way through all the hypotheticals from zero looks like a dead end for many practical scenarios, regardless of how powerful the model is. I think you could draw some analogies to humans here.

I have found that delegating deep research to a simple tool call is the best way to ground the agent in complex domains. If you make the main agent loop carry the weight of this research, it's going to do a really shitty job because of how the RLHF tries to preserve context and get an answer to the user quickly. As a tool, you may find the agent invokes multiple rounds of research consecutively without realizing it has incurred billions of tokens of consumption. Many of the tokens are wasted when generating independent hypotheses and subsequently investigating them, but the point is that you sampled 10-100x search space before getting serious about mutating the environment. The tradeoff seems worth it in a lot of cases. Correctness >> Time >> Money.

[SvenL]

Isn’t it weird to think a LLM “fully understands” something? What does this mean?

It reminds me of this printing trick “until you are 95% confident”…

Would really be interested to know how this influences the work of the LLM. Also, how would it be if I write “until you certainly understand it”?

[aszen]

Won't surprise me if the llm just calls sleep after it's convinced it knows all