[waterhouse]

I'm advocating that people do studies with randomized interventions—an approach that is capable of proving causation—and not being satisfied with purely observational studies that look at correlations, which are often hopelessly confounded and can't distinguish between "X causes Y" and "Z causes both X and Y". I was pleasantly surprised to find that this study was of the first type rather than the second.

[hombre_fatal]

While this take seems popular, this isn't a good way of looking at it because, in my experience, it seems to lead to the dismissal of good evidence or it suggests that we can't build confident causal inferences without a certain study. For example, RCTs are also observational, and it's incorrect to say they can uniquely detect causation.

I think a more helpful way to look at research is to look for a convergence of outcomes across the evidence, like a bunch of needles of various sizes pointing in the same direction (or not) on a gauge. And where there are divergences, and there always will be, which differences in methodology can explain them.

[waterhouse]

> RCTs are also observational

The key phrase is "purely observational". Now, occasionally you end up with a "natural experiment" in which some accident has effectively done the randomization for you—specifically, where the mechanism that puts people in the treatment group vs the control group is something you can be very confident has no other causal interactions. This was a good example: https://twitter.com/PGeldsetzer1/status/1661776663074738176

"Causal evidence that herpes zoster vaccination prevents a proportion of dementia cases [...] To provide causal as opposed to merely correlational evidence on this question, we take advantage of the fact that in Wales eligibility for the herpes zoster vaccine (Zostavax) for shingles prevention was determined based on an individual's exact date of birth. Those born before September 2 1933 were ineligible and remained ineligible for life, while those born on or after September 2 1933 were eligible to receive the vaccine."

But any time you're looking at a scenario where treatment vs non-treatment was the result of individual human choices, that opens up a potentially very wide range of ways for something you didn't know about (and potentially something difficult to accurately control for even if you do know about it) to cause treatment and cause the outcome, instead of the treatment causing the outcome.

> it suggests that we can't build confident causal inferences without a certain study

I do think there's an upper limit to the confidence you can justifiably hold, and that it's often not very high. Consider the studies that observe "A bit of alcohol correlates with better health than zero alcohol". You control for wealth, education, and maybe other things, and the apparent effect remains. How high confidence should you have in the result? Then someone realizes: Some fraction of people who consume zero alcohol do so under doctor's orders because they have health problems, and if you exclude those people then the effect disappears.

No matter how many causal pathways you think you've controlled for, how confident can you really be that there isn't a new one you haven't thought of? (And controlling has its own perils: if your measurements are noisy, or if you end up controlling for the outcome.)

> it's incorrect to say they can uniquely detect causation

Oh, correlational studies can "detect" causation, but the hard part is being certain that the thing detected isn't a false positive.

[TylerE]

And it'll also include some number of alcoholics in recovery who are currently consuming no alcohol, but who's bodies have hard years on them, and you'd expect to do worse than overall population average in outcomes.

[skybrian]

Yes, and any number of other things you hadn't thought of. The beauty of randomization is that, whatever it is, it will put some of those people in each group.

But even with a random variable, this still only gives you an average result for the group. And the more diverse the group is, the less likely that it's useful in predicting whether it works for you.

[LeifCarrotson]

I completely agree that randomized intervention studies have enormous value when compared to observational studies.

In this example, it would be particularly hard to establish a causative relationship that said "hearing aids inhibit dementia" or "hearing loss causes dementia" if instead all you had was a population, some of whom used hearing aids, and some of whom had varying levels of hearing abilities, and some of whom developed dementia. In all the noise, you'd be very fortunate to find that a control group of people with undiagnosed (how would you ever measure that!) hearing loss who failed to get hearing aids had cognitive decline in excess of that experienced by those who did get hearing aids. Here, instead, the intervention is "the coin flip said you're in the hearing aid group/you're in the health education group."

Of course, the murky ethical issue here is that this randomized intervention resulted in some people having poorer hearing, perhaps permanently (use it or lose it!) and some people having more rapid cognitive decline. This randomized intervention directly caused real suffering. Nearly 125 people in the heart-health group with increased risk of dementia onset were assigned to a study that just received health education. For at least three years, they had reduced hearing and a reduced ability to communicate, and after three years, they had significantly greater cognitive decline.

Caution is good, but in general I have no problem personally with the 'shut up and multiply' application of ethics in this case. There are 8 billion people all getting old, and more being born every day who will eventually get older. It's very important as a species to know that hearing aids slow cognitive decline, and if that meant that 119 people were placed at a slightly increased medical risk, 119 divided by 8 billion is a pretty small number, and 7.999999881 billion who are now more likely to get hearing aids and have slower cognitive decline is a lot of positive utility. But it wasn't me or my mom who can no longer hear and who is suffering from dementia, and some will draw a hard line and say that you can't perform studies that will put even a small number of people at increased risk.

We should be careful to evaluate merit and limit harm, but we should perform more randomized intervention studies.

[skybrian]

I largely agree, except that choosing a denominator of 8 billion is really dubious. The number of people for whom this study will result in them getting a hearing aid when they otherwise wouldn't is unknown, but unlikely to be nearly that high.

You can't actually do that math. Or rather, you can but the inputs to your spreadsheet are whatever you made up.

[LeifCarrotson]

Attempting to do the math is better than the alternative of going with your gut. The numerator is also an overestimate, not everyone suffered excessive cognitive decline.

If this was a critical decision (like it was for the authors of the study, unlike it is for the authors of Internet comments) you can work for better inputs.

Most importantly, though, we're trying to understand if that fraction is greater than one or less than one. There's a number that might be 8 billion or might be a tenth or a hundredth of that in the denominator, and a number that's on the order of 100 - or maybe 10x that, or maybe 1/10th that. But it doesn't matter, because it's not close, the ratio is still a million to one.