[KirinDave]

I agree with this, but we're not talking about equal weighting here. We are talking about absolute bias is towards no or yes without any inquiry.

The actual prior distribution of effective to ineffective models is extremely hard to infer from everyday life. We don't have access to unbiased data sources. That's why we should focus on inquiry rather than canned responses.

As an example of how this can be misleading common the average person sees only a tiny fraction of the proposed models that are much more likely to be valid because they've passed far enough along the process of publishing to have received some scrutiny and some credibility in the average case.

[darawk]

I think we probably actually agree. I am also opposed to any sort of inquiry-free canned response. I do think that, if you don't have time or desire to investigate, your baseline should still be that results like this are probably false. Even given the bit of investigation i've done into this study, and even though I think it's decently well done and reasonable, if I had to bet money on it, i'd bet that it's false.

However, I would say that if you don't have the time/desire to investigate, your comments in public forums probably aren't worth listening to. So, I do agree that people that simply respond with "correlation != causality" without reading the study probably shouldn't be doing that.

> As an example of how this can be misleading common the average person sees only a tiny fraction of the proposed models that are much more likely to be valid because they've passed far enough along the process of publishing to have received some scrutiny and some credibility in the average case.

Ya, that's definitely true. The base rate of true models in published research is almost tautologically higher than the base rate of true models amongst all possible models. I was going to say that it is tautological...but I suppose it's actually not. All published models could be false. But I certainly agree that the research process is a pretty good filter, and the base rate of true models is almost certainly higher. But i'd be very very surprised if it was higher than 50%, even if you use a fairly high standard for "published research".