[chatwinra]

"Its an experimental study, the number of subjects isn't really that important."

I'd be interested to understand why this is? My logical reaction would be that it's always important - as a crude example, surely doing an experiment on every single human on Earth would give you much more accurate results that on say 100 people, because you'd be sure to have covered all the innate variables that exist when experimenting with humans? (different metabolisms, etc)

[tel]

Depends on what kind of generality of statement you're trying to make. Here, the generality might be a cause and effect one in which case you're attempting to generalize over possible future treatments and you attack detractors who might yell "that was a fluke!" or "it wasn't the caffeine, but instead the presence of the doctor!". To do this, you design an experiment which carefully controls for all expected irrelevant interactions and then show a response which is significantly different from random variation.

You end up limited, as you note, to your population. 5 people won't defeat detractors who believe that this effect is limited within some, e.g., metabolic profile but it ought to give them serious food for thought as to how wide the affected metabolic profile actually is.

If these 5 volunteers were chosen at random, then the potential generality of effect can still be large as a detractor would be fighting, at best, with the notion that the 5 chosen were circumstantially susceptible to this effect (as compared to a study of convenience where one might believe that "college students" or "hospital volunteers" are especially susceptible).

So, in a certain sense, testing every human on earth improves the power of the statement you can make (not really its "accuracy" though maybe its "precision", in a sense), but in many other ways that may be too expensive for the kind of result the author seeks.

[MrJagil]

I have a feeling this test is not so much about accuracy as it is tendency. I.e. they avoid entirely specific projections, instead they want to see if the effect can be observed even once and thus have groundwork for further investigation.

[bildung]

>> "Its an experimental study, the number of subjects isn't really that important."

I'd be interested to understand why this is?

I replied to cossatot below in more detail. The short version: In studies like this one, N isn't 5, but humans(e.g. the original N) x treatment repetitions x measurement points.

You are right that it would be dangerous to ignore it, that wasn't what I implied.

[chaz72]

It'd be completely accurate, you would just need to keep every single human on Earth in controlled conditions. Good luck!