[maybenot]

This is a test of the default configurations of various brokers, NOT the brokers themselves.

Apart from declaring the testing queues in some brokers’ configuration and the persistence settings, all brokers were running with their default configuration out of the box (no tuning made).

He also casually mentions that ZeroMQ is using an in-memory configuration.

A home-made ZeroMQ 2.2.0 broker, working in memory only (no persistence).

The default configuration of AMQ persists messages to journal on disk. So apples to oranges?

Why do people keep writing "benchmark" blog posts like this?

[jdcryans]

My thoughts exactly, this is useless. Benchmarking is something everyone can do, so people do it, but it doesn't provide any real value unless you put an insane amount of time into it, and even then you'll get a lot of hate mail :)

For example, look at the "Yahoo! Cloud System Benchmark" AKA YCSB paper. Those researchers spent a lot of time designing the benchmarks and properly configuring each database that they were testing. They even dedicated a researcher per database to sit down with the developers to review the configurations and the test runs. I was part of this process for the Apache HBase as a dev. In the end, everyone was still critical of the results once they saw the graphs comparing the DBs.

But, I'm still glad they did it as I use YCSB as one of my tools to benchmark HBase.