[jroitgrund]

The coin example makes perfect sense to me and I had no idea what a p-value was before I read this article.

Here's an intuitive explanation: a p-value is the probability of getting your experimental results given that your hypothesis is wrong.

[Estragon]

> a p-value is the probability of getting your experimental results given that your hypothesis is wrong.

That's a common misconception. Actually it's the probability of getting your experimental results given that your null hypothesis is right.

[vlehto]

So "probability to perceive what you are perceiving just because of dumb luck"?

Then research with high p-value would just mean "we tried, and we still know practically nothing".

[im2w1l]

Well normally, hypothesis is wrong <-> null hypothesis is right.

[iamsohungry]

No, if the null hypothesis is right that implies that the hypothesis is wrong, but the implication relationship does not go the other direction.

Consider an experiment where your hypothesis is that cold temperatures cause the common cold. This is a good example for a thought experiment because we "know the answer" in a way (there have been a lot of experiments on this). The null hypothesis in this case is that cold temperatures are uncorrelated with incidence of the common cold.

You place people in isolation in cold areas and a control group in warm areas, and study how many get the common cold. None of the people who didn't already have colds get the common cold: because they are in isolation and the common cold is caused by rhinoviruses (which they can't get because they are in isolation), you get exactly the same results.

This disproves the hypothesis, but it does not prove the null hypothesis, that cold temperatures and the common cold are unrelated. Cold temperatures are, in fact, related to the common cold.

Try a second experiment: you place people in groups of five in cold areas and in warm areas, and discover a moderately high correlation between cold temperature and incidence of the common cold. This disproves the null hypothesis. But the simple hypothesis that cold temperature causes the common cold has also been disproven by your first experiment.

The reason for this is that the correlation between cold temperature and the common cold is a dependent correlation: given that rhinovirus is present in the system cold temperature is correlated with incidence of common cold (rhinoviruses reproduce ideally at temperatures significantly lower than human homeostatic temperature).

The null hypothesis is not just a statement that there is no independent correlation, it's a statement that there is no independent or dependent correlation. As such, the null hypothesis is an extremely broad hypothesis which is impossible to practically prove. This is why there's such a focus on finding correlations rather than finding non-correlations: you aren't going to prove the null hypothesis.

[bweitzman]

I think the coin example makes sense but is not a good example in this case. You will never be able to determine the validity of a coin just by flipping it. Any sequence of flips is technically valid but possibly just very rare for a fair coin.

The p-value won't tell you if the coin is fair or not, but it can tell you the probability that the coin is fair.

[skybrian]

If the sequence is rare enough then we do, in practice, conclude that the coin is unfair. If you can't make that conclusion then science doesn't work; no observations will convince you.

[bweitzman]

Of course. If you flip the coin 100 times and get all heads than you are safe to call the coin unfair, even though you would expect to see that result (1/2)^100 and thus would be wrong once in a while.

I think that's sort of the point that the article is making actually. That high probability does not imply truth. There are other non probabilistic ways to verify that coin is unfair, for example by looking at the density throughout the coin.

[TeMPOraL]

Observations do not imply truth. 0 and 1 are not probabilities, and in real life you can't prove something using observations in a logical sense. Real world runs on probabilities, not boolean logic.

Even if you look at the density of the metal throughout the coin, there's still a chance I've altered your device to report the coin is fair. Or a passing microsingularity decided to play games with the scanning beam. Or you're just imagining the whole thing.

That's not to say one should despair that the world is unknowable. One only has to get used to the fact that, in practice, "true" just means "extremely, extremely likely".

[bweitzman]

Sure, but there is a difference on the order of magnitudes between the probability that a fair coin will come up heads 100 times in a row and the probability that a microsingularity will come along and bias your results.

But yeah truth is tricky.

[tedsanders]

No no no. A p-value cannot tell you the probability a coin is fair. This is exactly the misconception that makes p-values a bad tool.

Suppose I have a coin and flip HHHHH. Can you tell me the probability the coin is fair? No, it's fundamentally unknowable. We can say that a fair coin would have a 3% chance of flipping HHHHH (the p-value), but we can't say with what probability our coin is fair.