[callahad]

Benchmarks are not comprehensive, nor are they gospel; it's entirely possible for a browser to score lower on a benchmark while also feeling faster in real-world use. For a long time, this was true of Chrome versus Firefox on JavaScript benchmarks: SpiderMonkey is routinely on par with V8, yet Firefox also felt significantly slower.

In the case of Speedometer, it's measuring one specific type of webapp interaction in a single tab; it's not capturing things like perceived speed of opening or closing the browser, switching tabs, loading pages, etc. It doesn't test how the browser performs when many JS-heavy tabs are loaded in the background. It also has lots of corner cases: installing a common password manager halves the score, and a common adblocker nearly does the same again, yet neither show that kind of noticeable degradation in real-world browsing.

Benchmarks are useful for saying "we've improved our browser this much in this area compared to where we were last year," but they're much less useful for making broad generalizations across heterogeneous browsers; there are too many other variables at play.

[Manishearth]

Whilst benchmarking stylo for example we noticed there were perceptible wins on interaction time on real world sites; but speedometer didn't budge as much because it tends towards small DOM trees which stylo can't parallelize much.