[slurry]

Why would you blame the study authors for doing a correlation study? I know "CORRELATION IS NOT CAUSATION!!!" is the go-to takedown on the internet, but correlation studies actually can have significant value - otherwise peer-reviewed journals wouldn't publish them.

I'm also skeptical of Politifact's competence to adjudicate public health scholarship.

[aamar]

I agree that correlation studies can be valuable, but I see many correlation studies as (intentionally or not) exploiting a propensity (arguably a bug) in human reasoning that conflates it with causation. In this situation, we have thorough documentation of multiple people clearly making the error.

I'm actually not sure the study authors should be blamed.

But: given how politicized this question is, they could have reasonably anticipated the misuse of their results, and thus could have written their results in such a way as to avoid this. Or they could have publicly corrected non-experts who cited them for causation. Or: they could have controlled for factors that would make mortality and insurance-status independent. This last option is difficult & requires complex judgement calls (see [1] for a reasonable attempt), but even if you feel the other two aren't required of academics, this last one very much may be.

(Separately, I agree that Politifact should not be trusted automatically, but these seem like reasonable analyses, and I did not quickly find anything better.)

[1] http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2739025/

[mattfenwick]

> correlation studies actually can have significant value - otherwise peer-reviewed journals wouldn't publish them

This is an appeal to authority. And unfortunately, the evidence is accumulating that there are problems with that authority. The success of peer-review depends on the quality of the peers and of the review. If those reviewing don't understand how to evaluate a correlation study, or do understand but don't take the time to properly evaluate it, then garbage will slip through.

It turns out, lots of garbage is produced and sent to the reviewers, as was noted in a recent Nature feature:

http://www.nature.com/news/scientific-method-statistical-err...

see also:

http://www.nature.com/news/weak-statistical-standards-implic...

http://www.nature.com/nature/focus/reproducibility/index.htm...

---------

tl,dr; lots of crappy correlation studies are published in peer-reviewed journals. These studies later turn out to be irreproducible.

[Fomite]

Not only do they have significant value, but they're the only type of evidence you'll ever have for something like this - it's impossible, and almost certainly unethical - to run a randomized trial of keeping people from having health insurance.

Also, while "Correlation is not causation" is, as you mentioned, a tired canard on the internet, people often seem to forget that all things that do have a causal relationship with have some form of association. Association doesn't prove causation, but it's a damned fine first step, and miles above "guessing", which is what happens without evidence.

[anatoly]

You keep saying 'unethical' as if it's the end of the discussion. But what if our current standard of medical ethics prevent us from finding the better policies or the better treatments? Is this standard set in stone and never to be doubted?

Probably taking away insurance from people who have it as an experiment is too extreme, and not implementable anyway. But is it really that unethical to take a subset of population without insurance, give it to a random subset for some time, and observe the differences? Why? No one from the control group is prevented from getting insurance on their own (compare with the candidate drug trials, where the control group can't just buy the drug on their own).

[Fomite]

> You keep saying 'unethical' as if it's the end of the discussion. But what if our current standard of medical ethics prevent us from finding the better policies or the better treatments? Is this standard set in stone and never to be doubted?

No, it's not set in stone, but if you're talking about implementing studies now, you're not going to get major medical ethics reform first. You go with the system you have, and the system you have is probably going to push back pretty hard.

> No one from the control group is prevented from getting insurance on their own

This alone is a difference between the control and treatment groups that takes place post randomization, and knocks said experiment back into the realm of "correlational"

[ekianjo]

Sorry, but no. 100 times no. Correlation is very often linked to a third factor or multiple factors which are not visible, nor measured in observational studies. Besides, let's not disregard the fact that correlation still has some good chance to be pure luck. Even correlation with 95% confidence statistical significance can be a random result in a non-nil number of times.

So, no, you never prove anything nor imply anything at all with correlation. You're still guessing.

[Fomite]

Find me where I said proof.

There are all kinds of things that we cannot prove, because it is either impossible or wildly unethical to conduct a randomized study. For those things, you can make a determined effort to control for as many "third factors" - the technical term for them is confounders - and that gives you a level of evidence which is well above guessing.

Since I can't reply to your comment, my responses here:

> "You didn't say proof but you said it's better than guessing, and I don't agree with you at all."

It is better than guessing. You're welcome to disagree, but a well conducted observational study is considerably firmer evidence than pulling it from your posterior.

> "What if there is a correlation between Vegetarian-lifestyle and Serial-killers ? Does it tell you that it's better than guessing ? Do you even question if the association/correlation makes remote sense ? Is there any underlying mechanism of action that would remotely explain rationally why this correlation could be linked to any real causation phenomenon ?"

All you've done is describe a really bad study. You can have really bad RCTs as well, by the way.

Of course you question whether or not an observed association has a clear biological or social mechanism. And you attempt to control for other variables that might influence the link between your exposure and your outcome. You run followup studies in different populations to try to understand if the result is a widespread phenomena, or a fleeting bit of statistical noise.

Basically, you do your job well. Which is why I used phrases like "a good first step".

Your example is about as useful as "Programming is useless because once I coded something poorly and corrupted my data".

[ekianjo]

You didn't say proof but you said it's better than guessing, and I don't agree with you at all. What if there is a correlation between Vegetarian-lifestyle and Serial-killers ? Does it tell you that it's better than guessing ? Do you even question if the association/correlation makes remote sense ? Is there any underlying mechanism of action that would remotely explain rationally why this correlation could be linked to any real causation phenomenon ?

Correlation is useless, and there's a ton of observational studies out there finding correlations every single day for which we have no rational explanation at all. Observational studies are full of variations in the way they are designed, the way they are reported and the subjects of the studies, it's rather a miracle if you actually detect a hint of causation based on the garbage noise that you get.

[mattfenwick]

> make a determined effort to control for as many "third factors"

How can you tell if you missed one?

> [...] gives you a level of evidence which is well above guessing

> It is better than guessing

Why? Without any support, this seems to be an appeal to probability.

> a well conducted observational study

> a really bad study

The fallacy of moving the goalposts; also the no true Scotsman fallacy.

[Fomite]

> How can you tell if you missed one?

Similar studies, using as many variables as you can find. Residual confounding is always and ever a problem, but the odds that something is both a strong residual confounder and has never been observed to have an association with the outcome or the exposure is pretty rare?

> Why? Without any support, this seems to be an appeal to probability.

It's really not - if for no other reason than it's forced you to think about your system more than a simple guess would. It's not an appeal to probability, its using data to update whatever prior you came in with. Guesswork is just using your prior.

> The fallacy of moving the goalposts; also the no true Scotsman fallacy.

Not really, no. Some observational studies are crap - this is just true. But that doesn't say anything about the potential quality of observational evidence, and many of the commonly raised objections to observational studies are actually objections to poorly run studies. The example used was a study that examined no potential confounding variables, looked at a correlation with no prior evidence suggesting any linkage between the two or biological plausibility, and then asserts that they've found a causal link.

That's a bad study. It's not 'No True Scotsman Fallacy' to say that the problems with a bad study don't generalize to all studies. If it is, then we're all screwed, because you can run a bad RCT too.

[mattfenwick]

> It's really not - if for no other reason than it's forced you to think about your system more than a simple guess would.

Appeal to probability; appeal to common sense.

> Some observational studies are crap - this is just true.

Special pleading.

> But that doesn't say anything about the potential quality of observational evidence

Straw man.

> many of the commonly raised objections to observational studies are actually objections to poorly run studies

> That's a bad study.

Moving the goalposts. No true Scotsman.

> It's not 'No True Scotsman Fallacy' to say that the problems with a bad study don't generalize to all studies.

Straw man.

[nzp]

> So, no, you never prove anything nor imply anything at all with correlation. You're still guessing.

This is a totally unreasonable stance to take. You can't even imply anything at all with correlation? Really, nothing at all? It's no better than a random guess? Try actually doing some actual science with this attitude, and keep to it consistently, and let me know how far you make it. In fact, try the same thing with ordinary life, any kind of reality where your decisions have consequences in reality.

[aaronbrethorst]

I'm also skeptical of Politifact's competence.

FTFY