[arkades]

The limiting factor on medical research isn't ethics: it's samples large and diverse enough to have a meaningful ability to distinguish between null and alternative hypotheses.

We produce shit studies with small sample sizes and dodgy methodologies all the time. They just don't add to the pool of human knowledge.

[AnthonyMouse]

> We produce shit studies with small sample sizes and dodgy methodologies all the time. They just don't add to the pool of human knowledge.

They actually do. A crappy study with p=.70 isn't enough to justify changing the standard treatment for a disease but it is enough to justify doing the more comprehensive study to see if the result still holds.

We have a choice between doing ten comprehensive studies only to prove that nine of the things didn't work, or using the same resources to do a hundred crappy studies and then repeat the five with the most promising results using the more robust methodology and discover that three of them still hold.

[arkades]

You confuse p-value with methodology. P-value is entirely a product of sample size and effect size. For a given effect size, the cheaper I want the study to be, the smaller I make the sample, and the less likely I am to reach a significant p-value.

The problem is when the methodology itself is broken. For instance, using case-control methodologies on things with small-to-moderate effect sizes. Past studies comparing such studies with eventual RCTs showed that the case control studies gave absolutely no indication of what the RCTs would show as a consensus: it was entirely random.

There is no amount of money, small or large, worth throwing at studies that shift our prior probability to posterior by 0 units.