[calvin]

Issue #1: performance claims based on tests from one device configuration. The best performance data is gathered from as many sources as possible. In the spectrum of statistics, one data point is not enough.

Issue #2: "Personally I have found the Peacekeeper results to be a reliable measurement of web browsers performance." Is there data to back up this claim?

[pcwalton]

I doubt Peacekeeper is representative of real Web pages. It primarily benchmarks things that are easy to measure. Peacekeeper also has several bugs:

* It accidentally benchmarks setTimeout clamping. https://bugzilla.mozilla.org/show_bug.cgi?id=610077 is a dependency of peacekeeper. https://bugzilla.mozilla.org/show_bug.cgi?id=608648 is an example in which setTimeout clamping affects the score a lot.

* Its benchmark of array.splice() is extremely strange: https://bugzilla.mozilla.org/show_bug.cgi?id=592786

* Its layout benchmarks do not really stress layout and instead either stress basic painting operations or DOM accessors. Most pages do not sit there calling style.top in a loop over and over.

* It sets MozTransform only in Firefox without setting values in Chrome: https://bugzilla.mozilla.org/show_bug.cgi?id=920659

And so on.

There is no reason to think that Peacekeeper's JS benchmarks are particularly better than V8's; in fact, they're probably worse, due to the proliferation of microbenchmarks. You'd get about the same effect by going to jsperf.com and clicking around.

[amelius]

Perhaps we should have meta-benchmarks.