[ucsdrake]

I would be really really careful with Phoronix benchmarking. If I recall correctly, I've seen them 'benchmark' and compare a FreeBSD system with a recent (at the time) Linux release. Issue was, they weren't even running the tests on the same hardware. <sarcasm> Really?! you saw performance differences between and i5 and i7?! Each running a different OS?! Get outa here!! </sarcasm>

[kbenson]

I'd be interested in seeing that. It's been a while since I really followed the site, but I seem to recall he was at least forward about what he was testing and why in the text around the benchmarks, I will admit to him seeming to be overly eager to get articles out, sometimes of ambiguous usefulness, probably due to his advertising model (at least at the time, when I was there earlier it seemed far less riddled with advertising).

Edit: Nevermind about the advertising, it's still intrusive, I'm just running adblock now.

[ryao]

Phoronix does try to be honest about its methodology, but the person running benchmarks gives no thought to what his results mean, if anything. It shows in the distinct lack of documentation on why a given benchmark matters.

As for Phoronix's ZFS benchmarks, the test hardware used drives that misreported themselves as having 512-byte sectors, which handicapped ZFS performance. Phoronix rejected all suggestions that it correct for this as end-users had been doing. Phoronix refused to meet half way by posting two results (one with proper configuration and one without), and also refused the suggestion that it to mention the existence of that problem in its test hardware. I eventually wrote code to identify drives known to misreport their sector sizes so that ZFS will automatically use the correct settings on them. That lead to the Phoronix August 2013 benchmarks showing a remarkable improvement in ZFS performance in FIO. It was so great that it sparked a discussion among the btrfs developers:

http://comments.gmane.org/gmane.comp.file-systems.btrfs/2754...

Later that month, I publicly criticized Phoronix for posting misleading benchmarks:

http://phoronix.com/forums/showthread.php?83731-ZFSOnLinux-0...

Phoronix has not posted ZFS benchmarks since that time.

[kbenson]

That's unfortunate, but doesn't surprise me all that much. Not because I think/thought of Larabel as likely to do something like that, but because I believe it's all too easy for organizations that are a single individual or organizations where a disproportionate amount of operation and decision making is really a single individual to make calls based on time and ability, and then fall back to defending that decision ever more vehemently long past the point where it should have been reassessed. Honest trusted feedback is priceless.