[Retric]

That XKCD is a joke and not how frequentists view statistics.

The confusion comes from the way you read papers. A frequentist looks at a paper as evidence but not truth. Bayesian on the other hand gets to the same place with slightly different math.

[marcosdumay]

The joke is about priors selection. I'd say it's spot on, since, as you say, the only reason people are not fooled like that is because they evaluate the frequentist results as evidence in an ad-hock Bayesian model.

The truth is that nobody thinks exclusively on frequentist or Bayesian terms. But that's not comics-grade material, and mixing them would hide instead of surface their differences.

[Retric]

The problem is it's a single sample. If the output was Yes, Yes, Yes, Yes, Yes, No, Yes, then you can do frequentist statistics, but when the output is just 'Yes' then the sample size is one.

[yummyfajitas]

There is no principled reason you can't compute a p-value from a sample size of 1.

[Retric]

What's the standard deviation of a sample size of one?

Now, the standard counter example is a composite statistic. Like roll 100 6 sided dice get 600 and assume they are not fair. But, importantly there is a standard deviation assumed in the experiment and there was more than one dice roll. However, if you combine that with something else then your sample size drops back to one.

[yummyfajitas]

The p-value is defined as p=P(evidence seen | null hypothesis). The standard deviation is only relevant if it is required to compute that number. You can run NHST on distributions without a p-value, e.g. a Cauchy distribution.

You might need a standard deviation if you want to do some naive Z-test based on the CLT approximation (since the normal distribution requires a standard deviation), but that's not what XKCD was describing. XKCD was describing an exact test using the true distribution.

[Retric]

True distribution is not what they where using. They assumed the dice had zero bias which is never true for any physical system.

I can say this is a dice and therefore it should have distribution X in theory. But, that does not mean it's actual distribution is X without testing. Further, even after testing nothing says the distribution will be unchanged.

Note: The above seem pedantic, but it has significant real world implications.

[raverbashing]

Having only one sample is not a problem if you're Bayesian.

[Retric]

Which is it's own problem. A Bayesian is often happy to look at any data that agrees with their own interpretation which is why it's not useful for papers.

The idea that A cause cancer is ridiculous. Collect data, well the A group has 10x as much cancer, but that's ridiculous so I conclude there is no relationship between A and cancer.

[red75prime]

Collect data, well the A group has 10x as much cancer, now it is about 10 times less ridiculous.

This should be about right.