[marcosdumay]

The joke is about priors selection. I'd say it's spot on, since, as you say, the only reason people are not fooled like that is because they evaluate the frequentist results as evidence in an ad-hock Bayesian model.

The truth is that nobody thinks exclusively on frequentist or Bayesian terms. But that's not comics-grade material, and mixing them would hide instead of surface their differences.

[Retric]

The problem is it's a single sample. If the output was Yes, Yes, Yes, Yes, Yes, No, Yes, then you can do frequentist statistics, but when the output is just 'Yes' then the sample size is one.

[yummyfajitas]

There is no principled reason you can't compute a p-value from a sample size of 1.

[Retric]

What's the standard deviation of a sample size of one?

Now, the standard counter example is a composite statistic. Like roll 100 6 sided dice get 600 and assume they are not fair. But, importantly there is a standard deviation assumed in the experiment and there was more than one dice roll. However, if you combine that with something else then your sample size drops back to one.

[yummyfajitas]

The p-value is defined as p=P(evidence seen | null hypothesis). The standard deviation is only relevant if it is required to compute that number. You can run NHST on distributions without a p-value, e.g. a Cauchy distribution.

You might need a standard deviation if you want to do some naive Z-test based on the CLT approximation (since the normal distribution requires a standard deviation), but that's not what XKCD was describing. XKCD was describing an exact test using the true distribution.

[Retric]

True distribution is not what they where using. They assumed the dice had zero bias which is never true for any physical system.

I can say this is a dice and therefore it should have distribution X in theory. But, that does not mean it's actual distribution is X without testing. Further, even after testing nothing says the distribution will be unchanged.

Note: The above seem pedantic, but it has significant real world implications.

[yummyfajitas]

All you're saying is models are imperfect. That's true. That doesn't mean you need a standard deviation or more than 1 sample.

In this case, the model of 1/36 odds of rolling 2x6 would have to actually be 1/20 (or smaller) to invalidate this test. Do you find it plausible that the bias in 2 die is that high?

[raverbashing]

Having only one sample is not a problem if you're Bayesian.

[Retric]

Which is it's own problem. A Bayesian is often happy to look at any data that agrees with their own interpretation which is why it's not useful for papers.

The idea that A cause cancer is ridiculous. Collect data, well the A group has 10x as much cancer, but that's ridiculous so I conclude there is no relationship between A and cancer.

[red75prime]

Collect data, well the A group has 10x as much cancer, now it is about 10 times less ridiculous.

This should be about right.