[shay_ker]

cost per benchmark task is definitely interesting!

i've always wanted cost per prompt, but even that has too much variation.

[janilowski]

Well yeah this benchmark I've quoted is quite large, I believe they use thousands of diverse tasks, so this average cost per task could be a more accurate representation of how expensive each model actually is to run.