[bel8]

With billions/trillions of dollars floating around, is it hard to imagine benchmarks could be biased?

I think it's safe to assume everything AI related is heavily biased until proven otherwise. Just like in pharma.

[camdenreslink]

People game benchmarks for fake internet points to get their favorite web framework to the top of the list. I'm pretty sure they will do it for billions of dollars.

[anthonypasq]

you didnt answer my question. Why would cognition be biased towards making anthropic look good?

[gloosx]

Because Cognition is a major customer of Anthropic?

[anthonypasq]

they are also a major customer of OpenAI and every other model maker. whats your point?