[evandijk70]

What effect size is big enough for you to worry about? A quick google search shows that smoking reduces life expectancy by 5-13 years depending on the study. Let's say that for smoking, the 'true' reduction of life expectancy from smoking is somewhere in the middle, 9 years. Then, the study I found with the 5-year estimate, is wrong by 4 years.

So, for some lifestyle change to reliably be reproduced by observational studies, it would have to be a 'true effect' of 4 years. I would say that 3 years, or even 1 year, is something to worry about.

[kmmlng]

Oh no, the evidence is clear when it comes to smoking. What I was trying to say: Smoking is a nice reference point to orient yourself around, because you can fairly confidently conclude that it's very harmful even with imperfect methods. So we can pick up on large negative effects using imperfect tools. At the same time, when you apply the same imperfect methods to other objects of study (e.g. the impact of coffee on health), you often get much more inconclusive results, which change from study to study. In that case, I don't think there is much sense in worrying until more robust methods can find clear evidence.