[scott_s]

Something did ping my radar, although it's hard to say because it's not published yet. What the news article says is:

But instead, the performance was largely similar, except when it came to the timing of events in the story. "The Kindle readers performed significantly worse on the plot reconstruction measure, ie, when they were asked to place 14 events in the correct order."

What I would like to know is: how many other performance measures did they test? How "significant" is "significantly worse"? If, say, they tested for 100 performance measures (unlikely, but I'm using a large number on purpose), then random chance means that there are likely to be some measures that are "significantly worse." If, on the other hand, they only tested 3 performance measures, then it's less likely to be random chance.

Basically, if you run an experiment and you test for a large number of things, you can't say much about the outliers. With large enough numbers, there are bound to be outliers. However, after you run such experiments, and you see those outliers, you can run more experiments to test if that was random chance, or if there really is some correlation there.

[seanflyon]

xkcd has a comic explaining the same thing.

http://xkcd.com/882/

[voronoff]

While the XKCD comic has a lot of truth to it, it's mainly about many different individual experiments (as well as some poorly done ones.) When running large sets of correlations, standard operating procedure is to use one of several techniques to counteract this effect.

[seanflyon]

Each performance measure is a different individual experiment.