- Ask it to update the skill to warn it against making that mistake again
I think that, using that iterative process, I end up with something better than just asking it to perform the task plainly
One observation: ask an LLM to write a skill, it tends to make them overly verbose and prescriptive. Often, something briefer and human-written actually works better
Another: I generally let the LLM propose edits to the skill, but I review them carefully and often modify them, because I find it has a tendency to solve the current problem at the price of worsening the solution to a previous one
> - Ask it to update the skill to warn it against making that mistake again
Yep! I have a canned prompt that basically boils down to "scroll up, read through and give me the top `n` things that were difficult..." and more often than not a small skill or change to `agents.md` comes out of that.
Technically speaking: it can't, any more than it can actually answer any other thing you ask it to do.
But in practice it does produce fairly reasonable output fairly often, and if you've been there watching and correcting it you can probably validate the result quite easily. "Re-read everything before" and similar are definitely anthropomorphizing, but that doesn't mean they're ineffective.
> One observation: ask an LLM to write a skill, it tends to make them overly verbose and prescriptive. Often, something briefer and human-written actually works better
Incredibly so. All too often a “hey, you shouldn’t have done that here” turns into “NEVER X”, without it actually understanding or making an effort to understand why the correction was made, though I would say usually the context to do so should suffice. I have a rule for it to reason this through and use nuanced language and would say the frontier GPTs (5.4 to 5.6 Sol) get it right around half of the time (very rough estimate). The conclusions are really dumb sometimes.
I’d be interested in an expertly crafted skill here, maybe there are things about it you can still improve. This is an area in which I still find LLMs to be quite lacking.
It also made me appreciate the complexity of the nuances and levels of indirection of what I want to teach it: “Upload to the remote host, which in this case is this but might be another one in another use case, only do this on the local system and this on the remote, do this to X unless Y …, prefer doing Z unless another rule overrides it …”
I sometimes feel like there could be a more expressive way to structure to structure these rules (bring the snark!).
Yep. That is what I do too. But a lot of times the changes it makes are bandaids rather than addressing root cause. I have to periodically evaluate the skills for consistency and opportunities for simplification.
Oh, skills are absolutely useful for giving the LLM distilled knowledge of things that it doesn't just know ("off the top of its head" so to speak). For example, a company's coding style guide is something that could be very useful to express as a skill.
My agent skills are just sets of instructions for my preferred way for something to be done. Mostly code reviews listing classes of bugs and output format and the like.
But also development instructions focusing on how and what to research before implementing (some of the stuff I do ends up heavily influenced by papers and publications while the naïve implementation is often bad, old, or gets stuck.
Not true. At work I have skills for different sets of APIs for different scenarios. That saves some prompt bloat because you rarely need to straddle projects. So you just load whatever you're using and don't have to maintain a a giant prompt.
They are useful for packaging all the related bits together. Some instructions on the line to add to the main prompt, detailed policy doc that spells out details, troubleshooting docs, the code itself, etc.
I actually don't mean that skills are useless, I have them in my projects too. Just that any time I've seen one shared publicly, basically the introductory blurb of "what does this skill do?" would serve adequately as "the skill". Such is the nature of machines that understand natural language.
It makes it kind of funny publishing skills online because by the time somebody knows they want the skill, they already have it.
Projects like SkillOpt[1] are pretty good evidence that the ceiling for skills isn't "describe it in a sentence and let the LLM generate it". There's value in iterating on a skill and figuring out what the weak parts are. At that point, sharing the skill can indeed be useful. Just look at superpowers, GSD, etc. They're 'just' skills (and a few bash scripts), but the text in them is useful enough to share.
I have something like "no over enthusiasm, and less marketing please. Also I have ADHD and therefore get distracted easily" in my agents.md. It made responses way more digestible and to the point.
But it's very useful and skills is a good way to share promots, so whatever.
Try the skill or the prompt. It will likely improve your experience interacting with agents