Hacker News new | ask | show | jobs
by gavinray 58 days ago 

  > Letting the LLM write half baked tools is the recipe for burning more tokens.
It sure is, if the tools are half-baked and your user scale is N=1 rather than N=100 or N=1,000

  > What's the criteria for marking an LLM written tool as useful/correct before publishing it?
It solves the problem the originating user asked it to
2 comments
afshinmeh 58 days ago
> It solves the problem the originating user asked it to

Interesting. And is there a mechanism to go back and "fix" the tools after they are published? What happens if the tool decided to use the "id" attribute to click on buttons and now you have a new website that follows a different pattern to find the right target?

I agree that "correctness" of a tool could have different meaning depending on the context of the problem though (e.g. would you consider OOM a correctness bug even if it addresses the user's ask?)

link
sifar 58 days ago
The problem here is that N different users will ask for N different variants of the same tool, so you'll end up with a tool which is similar but not quite. Is the tool updated to support new functionality, or a new tool is created and you end up with N variants of a tool.
link