[pas]

Doesn't this also imply that the effect is "small" even if real? After all if there would be a very strong correlation (eg a true and exclusive casual chain) then we would see a huge signal even in small and noisy datasets. Or am I missing something?

[BenoitEssiambre]

No. If you look closely at the data you might be able to draw such conclusions but lack of statistical significance often doesn't suggest or imply a small effect. Notice that in this case in particular, the confidence intervals are consistent with very large positive or fairly large negative effects. Don't underestimate the amount of noise often found in studies. Lack of significance usually just means that data is too noisy to tell us anything. If you get significance you get to say: the data is probably not pure noise but the effect could still be very tiny or caused by systematic measurement errors. Null hypothesis testing is pretty useless really.

[pas]

Thanks for your detailed reply! I meant that even this dataset puts a limit on the effect size, if viewed as an "evidence of absence of clear and large effect".

Of course since "everything is correlated" [0] expecting such truly simple signals might be nonsensical/pointless.

I was just lamenting the lack of simple magical treatments basically.

[0] https://www.gwern.net/Everything

[BenoitEssiambre]

That gwern page is excellent!