Hacker News new | ask | show | jobs
by hervature 1737 days ago
It is a paired t-test as the test takers are the same individuals. This is confirmed in the statistical analysis section of the actual paper. Paired t-test have more statistical power because we don't correct for group variation (as they are the same group measured twice).

Edit: One way to think about this is assuming the effects are random and symmetric then the probability of going up is 1/2. If all 6 participants had increased scores, then the p-value is at least 1/2^6 and this is not taking into account the magnitude of the shift.

1 comments

You're right about the paired test (although I think there's an issue with the group variance). But your final calculation is for a sign test. For a t-test where every point shifts a very little bit (e.g. 0.01 stddev), p is larger, I think.
Yes, my edit was more meant to illustrate how 6 data points can yield a significant result. Paired t-test is all about the differences. So if all test subjects shift up by 0.01 (no variance in the differences) then the statistic will actually be infinite (i.e. very large) and so p is theoretically 0 (very small). I also agree that the paired testing needs more convincing from the researchers.