Hacker News new | ask | show | jobs
by andrioni 2936 days ago
This kind of clause is usually known as a "DeWitt clause"[1], after a University of Wisconsin professor who benchmarked a couple databases, including Oracle (which performed poorly). Oracle/Larry Ellison didn't react well to that and decided to forbid benchmarks.

[1]: https://danluu.com/anon-benchmark/
3 comments
JoeAltmaier 2936 days ago
Database performance depends on so many variables - thread pools, queue sizes, RAM allocations to different purposes, disk layout - the list goes on. There are whole consulting fields that are essentially doing a random walk through database configuration files looking for better performance - ascending the performance gradient if you will.

So in this world, its sensible to say "benchmarks are uninformative and misleading - your mileage will almost certainly vary"

link
solussd 2936 days ago
Benchmarks are easily abused, misused, and misinterpreted. E.g., benchmarks looking at some very specific aspect of query performance being extrapolated to more complex/real-world queries.

Also trade-offs are rarely mentioned in benchmark numbers– e.g., great write throughput, at the expense of: ?.

It's fun to be cynical about stuff like this, but it's rarely as simple as "Ellison didn't react well to that and decided to forbid benchmarks".

link
alter_eco123 2936 days ago
> Oracle/Larry Ellison didn't react well to that and decided to forbid benchmarks.

So you kind of have to wonder why Cognitect is going Oracle on us..

The most obvious explanation is that Datomic just doesn't perform well and they don't want people to know.

link
_halgari 2935 days ago
Anyone who has done serious performance testing on a DB knows that there's a massive gap between initial findings and a well tuned system designed with the help of the database maintainers. I've seen some nasty performance out of Riak, Cassandra, SQL, ElasticSearch etc. But with each of those, once I talked to the DB owners and fully understood the limitations of the system it was possible to make massive gains in performance.

Databases are complex programs, and if I ever wrote one, it would be infuriating for someone to pick it up, assume it was "just like MySQL" and then write a blog post crapping on it because it failed to meet their expectations.

link
alter_eco123 2935 days ago
Yes, benchmarks can give a misleading impression of a database's performance.

So what? Somehow PostgreSQL is doing fine despite that.

Which is worse publicity for Cognitect: people publishing bad benchmarks or Cognitect forbidding benchmarks Oracle style?

link