Hacker News new | ask | show | jobs
by tyingq 1679 days ago
I read the original post, the Snowflake response, and this. From that I gather that both of them aren't being completely honest or fair when making comparisons. A fair amount of truth, but also some clever wording and omission on both their parts. Which is not surprising or particularly new in this space :)
1 comments
slownews45 1679 days ago
Databricks results are available at tpc.org [1]

Snowflake has shown NOTHING close to this.

[1] http://tpc.org/results/fdr/tpcds/databricks~tpcds~100000~dat...

link
david_allison 1679 days ago
Sorry to nitpick (document seems solid), on page 32:

> Due to a TPC-internal error during the production of 3.2.0 of the TPC-DS kit, the benchmark execution had to use version 2.13 of the kit. It was confirmed by the TPC that the only changes between these two versions of the kit is the version number set in the tools/release.h parameter file.

How can there be that much of a delta of major/minor versions without a change? The only way that I see this happening is if 'change' being defined as the specific benchmark which was run, rather than the kit.

link
ni_po 1679 days ago
The change is in the Spec ie., allowing cloud storage and the metric itself causing the major version update, not in the datagen binaries.
link
tyingq 1679 days ago
Yes, I wasn't saying they were lying about their tpc.org posted results. I'm saying both companies made use of clever indirection, wording, presentations of stats, etc. Like price/performance, and which of your competitor's tier's to select when doing that, and which of your own. Or over-provisioning the competition's setup, for example.
link
slownews45 1679 days ago
Ahh, fair enough there. That said, snowflake would help their case if they would actually do at least one actual tpc.org third party result
link
tyingq 1679 days ago
My guess is that they know the result won't look terrific. And they also know Snowflake works well in production for people despite that. So, little upside.
link
slownews45 1679 days ago
For sure.

All leaders in a space take this approach. Little be gained, a fair bit to lose if you are ALREADY leading without having to debate / do a benchmark etc.

Anyways, the benchmark is only one part of the overall story for these solutions.

link