Hacker News new | ask | show | jobs
by yreg 22 days ago
Yeah, if the benchmark is actually predictive of the tasks you have then it is trivial to conclude that the cheapest-per-benchmark-task model will be the cheapest one for your tasks…
1 comments

It might vary between tasks though. A model that’s great at abstract reasoning might be great at writing math proofs but struggle to write software in <insert language>.