[tpoacher]

Depends. Emotive language aside (i.e. "damning") the low sample size is actually "more evidence" in the presence of a convincingly low p value.

It means that this particular sample of 21 people would have to be extremely atypical under the null hypothesis of no difference. Therefore, given its low sample size (and thus wide standard error), it is 'some' evidence that the effect size is also probably quite large.

[majormajor]

That doesn't sound right. The reason sample size is important is because the larger the group, the less likely the group is to be atypical. I don't think you can turn that around and then say "this group is small, which itself is evidence, because otherwise it would have to be extremely atypical to show a difference." That's completely circular.

The p-value is still the important thing because the p-value is itself affected by the sample size.

[stevenhuang]

No, effect size very much informs all of that.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3444174/

[tpoacher]

> the larger the group, the less likely the group is to be atypical.

Not true (unless I misunderstood what you were trying too say). The probability that a sample will be atypical under the null is exactly the significance level, and does not depend on its size.

What does change with size is that the distribution around the statistic gets more concentrated, meaning your threshold moves to the left, towards lower effect sizes.

Which also means that for smaller sample sizes, it is harder to reach significance, unless you're dealing with a fairly large effect size.