[barrkel]

Benchmarks tell me that Firefox's throughput is better at this point, and I have no reason to doubt it, but I also know from direct experience that variance in performance is far, far higher with Firefox.

I don't think I've ever run any asm.js other than demos.

PS: I just ran Octane 2.0. The SplayLatency test looks like it's designed to test what I'm talking about; I get 25k in Chrome, but only 9k in FF. http://developers.google.com/octane/benchmark#splaylatency

A bunch of people reckon it doesn't matter - see e.g. https://news.ycombinator.com/item?id=8519507 - but I think it does. GC pauses are very noticeable when there are realtime constraints, like animation.

[azakai]

Yet in the other latency test in Octane, MandreelLatency, Firefox does about 50% better than Chrome.

Perceived performance depends on a lot of things, not just JS speed and consistency - also graphics, UI responsiveness, etc. Chrome has had an advantage in some of those areas, but the difference is pretty small at this point, especially if you are on Firefox Nightly.

[cromwellian]

Isn't Mandreel just another Emscripten style test that avoids the garbage collector in favor of typed arrays?

I think we definitely need some GC throughput/latency benchmarks in the standard suites.

[azakai]

Mandreel doesn't avoid garbage as much as emscripten output does, but it does create less than "typical" JS, most likely. However, it still can create plenty of doubles as garbage on the heap, on JS engines without nanboxing, such as v8, which is a factor in why v8 loses here.

[bzbarsky]

Here's the problem with SplayLatency.

SplayLatency computes the root-mean-squared allocation time, and your final score is the reciprocal of that, scaled by some constant. Let's call the RMS measurement the "badness"; more badness is worse on this benchmark.

Say you're allocating 1000 objects and object allocation itself takes 0 time so all that's being measured is the GC. You plan to GC them all before your JS runs to completion. You consider two alternate strategies.

One strategy is to perform a GC run every 10 allocations, each of which collects 10 objects. For simplicity, say each GC has 1s of overhead and takes 1s per object collected. So each of your GCs will take 11s. So you will have sqrt((100 * 11^2 + 900 * 0)/1000) = sqrt(12.1) as your "badness" score on the benchmark, and will take 1100s to finish.

Now the second strategy: one GC every 100 allocations. Now each GC takes 101s, and it only takes 1010s to finish. But the splay "badness" score is sqrt((10 * 101^2 + 990 * 0)/1000) = sqrt(102.01).

So per the benchmark the better strategy is the "GC more often" one. But for animations the _second_ strategy is better in this case, because the animations can't run while the JS is running to completion anyway. So as long as both strategies are collecting all the garbage before run-to-completion finishes, the one that's better for animations is the one with higher throughput. But that's the one Splay scores worse.

Back to the real problem we're trying to solve: what hurts animations is a GC strategy that aims for higher throughput by letting garbage pile up across multiple runs to completion and then ends up with a long GC pause at some point. Having a benchmark that penalized that sort of GC strategy would in fact be a good idea. But Splay is not that benchmark. In fact, the optimal GC strategy on Splay is to not GC at all until the benchmark finishes and then do one big GC that takes forever but isn't measured as part of the benchmark time.

Basically, SplayLatency sets up perverse incentives where the simplest ways to do better on the benchmark involve making animation pauses _worse_.

It's also possible to improve the score on SplayLatency by actually improving the throughput of your GC, but that's a lot more work than the other approaches, and just as likely to regress this benchmark if you do it by chunking your GC more within a single run to completion.

The end result is that improvements to this benchmark's score have little to do with reduction of user-visible GC pauses.

You can see some more in-depth discussion in https://bugzilla.mozilla.org/show_bug.cgi?id=958492 but the above basically summarizes what's going on. The fix in that bug ended up just shuffling work around within a single run to completion to placate this benchmark, and the hard part was doing it in a way that didn't regress things too much for actual real-life animation use cases...