[matt4077]

People love the "n=XX is far too little data!" argument, yet it's more complicated than that. Sometimes 600,000 is too little, yet sometimes 17 is enough.

Example: you believe a newly found plant species is toxic. You give it to 17 "grad students volunteers", while giving a placebo to 17 others. All in the first group die aa gruesome death within 20 hours. None of the others do.

Result: yes significance. (also: tenure!)

I'm not saying that this study is significant (the statistics seem to be slightly beyond my event horizon), and your criticism also stops short of an outright dismissal of the research. But sample size alone makes for a bad measure of quality. Yes, even p-values are better.

[onetwotree]

I think that a small sample size is mostly an indicator that one needs to treat the results with far greater caution.

Effect size is very important in this. To continue your grad student murder example, it's completely trivial to determine which plant a student was given, based on whether they are dead or not. It becomes trickier if you measured something a bit less cut-and-dry, such as the incidence of headaches, or variance in a few voxels of a noisy MRI.

[jacalata]

I really hope you wouldn't get tenure for a study that killed all your subjects.

[rav]

Subjects? But they're volunteer grad students!

[jacalata]

Subjects, minions, whatever you want to call them :p

[hikarudo]

The technical term for that is "effect size".