[6thbit]

Anyone has an insight into how much money labs are putting into benchmarks?

Just Arg-AGI-3 is quoted above 20K USD and footnote says average of 5 runs (!!). Likely just a drop in the bucket to the training budget but still..

[jofzar]

It's $0/close to 0, they aren't at 100% demand so any leftover compute is "not spent".

The cost they are quoting is API cost, so it's already inflated on that.

[stri8ted]

20k is small potatoes for the marketing impact.