[Splines]

Not to discredit their results, but what it shows is that users who used Edge consumed less power than Chrome or Firefox.

Whether that means Edge is more power efficient, or devices where Edge is used more often are more power efficient, or users who select for Edge tend to visit sites that allow Edge to be more power efficient isn't clear.

To be fair to both sides, this is a really hard thing to measure.

[AnthonyMouse]

> To be fair to both sides, this is a really hard thing to measure.

Is it though? It's more the fact that Big Data is totally useless because it contains thousands of uncontrolled variables that confound the results.

If you really want to know which browser is more power efficient, get a sample of the commonly visited sites and then visit the same sites on the same common devices in different browsers and see which one uses less power.

You can still get the wrong outcome by using a biased sample on purpose, but then the outcome is wrong because you fudged it, not because it's hard to get the real answer.

[tgb]

That's what opera just did!

[admax88q]

That's also what Microsoft did.

It's quite possible that both results are valid. It sounds like there's big difference in the sites that Microsoft visited compared to Opera.

It seems an even better methodology would be to record actual user browsing sessions, and then replay them through different browsers and measure power consumption.

Opera's simulated browsing session does not sound particularly sophisticated IMO.

[Aelinsaar]

Couldn't you just make a giant virtual "browsing session" with hundreds of thousands of different sites , and crunch the numbers to minimize the background noise? MS has the computing power, that's for sure.

[bzbarsky]

Except that this has an interesting failure mode: unless you're visiting the sites by hand you're presumably using some browser automation framework, with different implementations for different browsers. And _those_ can have nontrivial power usage, depending on how they work. I know for a fact some of them do power-hungry continuous polling...

[dhaivatpandya]

It seems pretty unlikely that the data wouldn't be normalized by some metric of intensity of use since this is pretty much the first thing anyone would think of. Normalizing even by something as simple as "number of websites visited" would, I imagine, significantly reduce the contribution of such factors to the differences observed.

[matmann2001]

Or they just ignore normalization, since it's for PR and not an actual scientific study meant to withstand scrutiny.

[harigov]

The fact that they spent so much money in hardware to test the battery usage and the amount of effort they put in to improve doesn't seem to imply that this was just a PR stunt.

[matmann2001]

You underestimate the value attributed to PR by large corporations.

[KirinDave]

> Not to discredit their results, but what it shows is that users who used Edge consumed less power than Chrome or Firefox.

The workloads Microsoft simulated were held constant for every platform and instrumented via a common command-and-control testing library. It is very much apples-to-apples. The additional telemetry they released is even more confirmation.

Opera replied by enabling a pair of special default-off features, both of which significantly change the workload and the user experience for their browser, put it on a random laptop pair which might not even be from the same lot or subtly different models, and then made a timelapse.

[Splines]

> The workloads Microsoft simulated were held constant for every platform and instrumented via a common command-and-control testing library. It is very much apples-to-apples.

I'm not doubting this part at all.

> The additional telemetry they released is even more confirmation.

The additional telemetry doesn't really tell much of a story either way, given that it's presented as a single chart and little context is provided. Unfortunately there's no link to the data analysis to understand what factors were considered when looking at the data sets involved.

Personally I've found that it's hard to come to these high-level conclusions about performance from large data sets, because users and their devices are so incredibly diverse. It's really hard to draw definitive conclusions given all the data.

It's dangerous to take your data, aggregate it at a high level and look at the results from that point of view - you lose the context under which it was gathered and can come to the wrong answer (for example, browser X is popular in a country with great internet, so page loads are faster, and browser Y is popular in a country with terrible internet, so page loads are slower).