Hacker News new | ask | show | jobs
by OtherShrezzing 20 days ago
>4x improvement on geospatial tasks with map in the loop.

The graph shows a baseline 2% task success rate improving to to 8% task success rate, but the evals section details 100% success rates across the board.

I'm not sure what the effectiveness of this skill is from the readme. Is it 8% success, or 100% success?

1 comments

It's not % it's number of evals. Here is how I did evaluation https://dekart.xyz/blog/how-to-evaluate-claude-skill-output-...