[Lerc]

That does seem to be a concern here. They are replacing Octane with what amounts to a-collection-of-stuff. Without a specific workset (ie. a benchmark) there is no clear way to measure one thing against another.

This seems like a mechanism to hide where you stand. Now they can just say "That's not what we find in our tests".

I'm inclined to agree that becnhmarks do become obsolete as workloads change and development targets benchmark pointscoring. The solution is to make a new standardised benchmark, or indeed many of them. If they are clear about what they are measuring then the relative perfomance of different JavaScript engines can be understood by developers to give a nuanced view of what performs well.

[codinghorror]

We find that http://browserbench.org/Speedometer/ -- which is what v8 is optimizing against now -- is extremely close to real world Discourse (EmberJS, Glimmer) performance.

We always wondered why Android was so crazy ridiculously slow at our real world JS vs. iOS starting in late 2013 onward.. getting worse and worse over 2014 and 2015.. and "gee, we optimized for the wrong thing aka Octane" was a big part of that.

http://benediktmeurer.de/2016/12/16/the-truth-about-traditio...

You can read more about the history at https://meta.discourse.org/t/the-state-of-javascript-on-andr...

[om2]

JavaScriptCore (WebKit's JS engine) also optimized for Octane. But we additionally optimized for Speedometer, Sunspider (the not totally broken subset that made it into Jetstream) and JSBench. While Octane isn't the single answer to all JS performance measurement, using it in combination with better things seems better than not using it. To go back to your example, iOS both has way better real-world perf and way better Octane results than Android devices.

BTW benchmarks I'd endorse include MotionMark, Speedometer and Jetstream from http://browserbench.org/ as well as JSBench http://plg.uwaterloo.ca/~dynjs/jsbench/

These are all good at testing different aspects of browser performance including relevant testing of the JS engine.