[kmmlng]

The fact that you can pick up on negative effects through observational studies if they are large enough makes me especially suspicious about results that differ from study to study. Perhaps there is an effect there, but if we can't reliably pick up a signal, maybe it's not something I need to worry about.

[evandijk70]

What effect size is big enough for you to worry about? A quick google search shows that smoking reduces life expectancy by 5-13 years depending on the study. Let's say that for smoking, the 'true' reduction of life expectancy from smoking is somewhere in the middle, 9 years. Then, the study I found with the 5-year estimate, is wrong by 4 years.

So, for some lifestyle change to reliably be reproduced by observational studies, it would have to be a 'true effect' of 4 years. I would say that 3 years, or even 1 year, is something to worry about.

[kmmlng]

Oh no, the evidence is clear when it comes to smoking. What I was trying to say: Smoking is a nice reference point to orient yourself around, because you can fairly confidently conclude that it's very harmful even with imperfect methods. So we can pick up on large negative effects using imperfect tools. At the same time, when you apply the same imperfect methods to other objects of study (e.g. the impact of coffee on health), you often get much more inconclusive results, which change from study to study. In that case, I don't think there is much sense in worrying until more robust methods can find clear evidence.

[sjducb]

Agreed.

You can’t prove a negative, but you can define limits on the effect size.

You can show that the maximum possible size effect size is so small that you’re not going to worry about it.