Hacker News new | ask | show | jobs
by karmasimida 1048 days ago
On HumanEval, Copilot is 40+ on pass@1 comparing to 26 for stable code 3b.

HumanEval is abused but this model is only good for its size, it is no match for Copilot … yet
1 comments
UncleOxidant 1048 days ago
> On HumanEval, Copilot is 40+ on pass@1 comparing to 26 for stable code 3b.

Can you put those numbers into context for those who haven't done HumanEval? Are those percentages so that 40+ means 40+% and 26 is 26%? If so does that imply both would be failing scores?

link