[lmeyerov]

It sounds fundamentally busted if a competitor can submit benchmarks for someone else. TPC is great in general, but I didn't realize it had such a gaping flaw.

TPC submissions take real time/$/energy/expertise, so I don't know anyone who has ever done it casually. Ex: It was a multi-company effort for the RAPIDS community to get enough API coverage & edge case optimization for an end-to-end GPU submission on the big data one (SQL, ...), and even there the TPC folks made them resubmit if I remember right.

Also, note how the parent's response did not actually answer 'audited how'. Pushing the work to the questioner is on the shortlist of techniques studied by misinformation researchers. I'm a fan of both companies, so disappointing to see from a company rep.

[rxin]

Check my reply, Leo.

[lmeyerov]

The audit question is on Databricks marketing unaudited Snowflake TPC numbers. I do think Snowflake is big enough to run TPC, but how you guys choose to market is on you.

But: I think it's cool both companies got it to $200-300. Way better than years ago. Next stop: GPUs :)

[rxin]

Ah ok. Wasn't clear. I think some repro scripts will be available soon.