[hombre_fatal]

While this take seems popular, this isn't a good way of looking at it because, in my experience, it seems to lead to the dismissal of good evidence or it suggests that we can't build confident causal inferences without a certain study. For example, RCTs are also observational, and it's incorrect to say they can uniquely detect causation.

I think a more helpful way to look at research is to look for a convergence of outcomes across the evidence, like a bunch of needles of various sizes pointing in the same direction (or not) on a gauge. And where there are divergences, and there always will be, which differences in methodology can explain them.

[waterhouse]

> RCTs are also observational

The key phrase is "purely observational". Now, occasionally you end up with a "natural experiment" in which some accident has effectively done the randomization for you—specifically, where the mechanism that puts people in the treatment group vs the control group is something you can be very confident has no other causal interactions. This was a good example: https://twitter.com/PGeldsetzer1/status/1661776663074738176

"Causal evidence that herpes zoster vaccination prevents a proportion of dementia cases [...] To provide causal as opposed to merely correlational evidence on this question, we take advantage of the fact that in Wales eligibility for the herpes zoster vaccine (Zostavax) for shingles prevention was determined based on an individual's exact date of birth. Those born before September 2 1933 were ineligible and remained ineligible for life, while those born on or after September 2 1933 were eligible to receive the vaccine."

But any time you're looking at a scenario where treatment vs non-treatment was the result of individual human choices, that opens up a potentially very wide range of ways for something you didn't know about (and potentially something difficult to accurately control for even if you do know about it) to cause treatment and cause the outcome, instead of the treatment causing the outcome.

> it suggests that we can't build confident causal inferences without a certain study

I do think there's an upper limit to the confidence you can justifiably hold, and that it's often not very high. Consider the studies that observe "A bit of alcohol correlates with better health than zero alcohol". You control for wealth, education, and maybe other things, and the apparent effect remains. How high confidence should you have in the result? Then someone realizes: Some fraction of people who consume zero alcohol do so under doctor's orders because they have health problems, and if you exclude those people then the effect disappears.

No matter how many causal pathways you think you've controlled for, how confident can you really be that there isn't a new one you haven't thought of? (And controlling has its own perils: if your measurements are noisy, or if you end up controlling for the outcome.)

> it's incorrect to say they can uniquely detect causation

Oh, correlational studies can "detect" causation, but the hard part is being certain that the thing detected isn't a false positive.

[TylerE]

And it'll also include some number of alcoholics in recovery who are currently consuming no alcohol, but who's bodies have hard years on them, and you'd expect to do worse than overall population average in outcomes.

[skybrian]

Yes, and any number of other things you hadn't thought of. The beauty of randomization is that, whatever it is, it will put some of those people in each group.

But even with a random variable, this still only gives you an average result for the group. And the more diverse the group is, the less likely that it's useful in predicting whether it works for you.