[jiggawatts]

Something I find odd in the AI space is that almost all journalists republish vendor benchmark claims without question.

Why not just benchmark the models yourself?

Tiny little YouTube channels will spend weeks benchmarking every motherboard from every manufacturer to detect even the tiniest differences!

Car reviews will often test drive the cars and run their own dyno tests.

Etc…

AI reviews meanwhile are just copy-paste from the market blurb.

[latchkey]

Even the 3rd party AI benchmarks that are published [0], are all sham too. It is run by a paid shill (semianalysis) and all highly tuned by the vendors to make themselves look good.

[0] https://github.com/InferenceMAX/InferenceMAX/

[coldtea]

>Why not just benchmark the models yourself?

Because their incentives are to churn stupid articles fast to get more views, and to be on major AI companies and potential advertisers' good graces. That, and their integrity and passion for what they do is minimal, plus they're paid peanuts.

Doesn't help that most brain-rotted readers are hardly calling them out for it, if they even notice it.

[CamouflagedKiwi]

It's not free to run those benchmarks, especially on the big models.

Ideally journalists / their employers would swallow that as the cost of business, but it's a hard sell if they are feeling the squeeze and aren't making much in the first place.