[Seanzie]

For as long as journals refuse to publish negative results, researchers will scrounge for some type of positive result, even if it's causally questionable, because to do otherwise is to see years of work go for nought. Until these incentives change, the rest of us are right to be skeptical.

[KirinDave]

Skepticism is about inquiry, not denialism. You're not practicing skepticism if you make a decision that something is false. The point is to defer decision (or threshold your decision probabilisticly) on the evidence as it becomes available.

Saying "no" to everything is no different than saying "yes" to everything from a logical perspective.

[darawk]

> Saying "no" to everything is no different than saying "yes" to everything from a logical perspective.

This is incorrect. The base rate of true findings when talking about causal models is extremely low. If you say "no" to every published finding, you will be right much more often than you're wrong.

Now, that doesn't mean you should say no to every published finding. But the idea that "yes" and "no" should be equally weighted in your priors across the board is an inaccurate representation of the state of research, and the underlying facts of the universe.

[KirinDave]

I agree with this, but we're not talking about equal weighting here. We are talking about absolute bias is towards no or yes without any inquiry.

The actual prior distribution of effective to ineffective models is extremely hard to infer from everyday life. We don't have access to unbiased data sources. That's why we should focus on inquiry rather than canned responses.

As an example of how this can be misleading common the average person sees only a tiny fraction of the proposed models that are much more likely to be valid because they've passed far enough along the process of publishing to have received some scrutiny and some credibility in the average case.

[darawk]

I think we probably actually agree. I am also opposed to any sort of inquiry-free canned response. I do think that, if you don't have time or desire to investigate, your baseline should still be that results like this are probably false. Even given the bit of investigation i've done into this study, and even though I think it's decently well done and reasonable, if I had to bet money on it, i'd bet that it's false.

However, I would say that if you don't have the time/desire to investigate, your comments in public forums probably aren't worth listening to. So, I do agree that people that simply respond with "correlation != causality" without reading the study probably shouldn't be doing that.

> As an example of how this can be misleading common the average person sees only a tiny fraction of the proposed models that are much more likely to be valid because they've passed far enough along the process of publishing to have received some scrutiny and some credibility in the average case.

Ya, that's definitely true. The base rate of true models in published research is almost tautologically higher than the base rate of true models amongst all possible models. I was going to say that it is tautological...but I suppose it's actually not. All published models could be false. But I certainly agree that the research process is a pretty good filter, and the base rate of true models is almost certainly higher. But i'd be very very surprised if it was higher than 50%, even if you use a fairly high standard for "published research".

[L2R]

Researcher here (in ML), our field is so full of noisy results due to this issue. Everyone talks about it, but you can't get around the fact that you no longer can get away with a low amount of publications.

[yachtman]

I dont see why journals can't publish more negative results. In math for example, its a big deal if someone disproves a hypothesis of someone famous.

[jefftk]

The math equivalent of a negative result would be more like "I tried for months to prove this theorem, and it didn't work".

The difference is that in math "here is a proof" is all you need, while in sciences people are typically sharing what amounts to collections of observations. If you publish the collections that lead to interesting conclusions, but not the ones that are boring, people looking at what's been published are misled.

[dwaltrip]

A negative result means simply they didn't find compelling evidence for their hypothesis. Nothing is proven or disproven (although it adds weight to the idea that the hypothesis is not true), so people think it is unexciting. For that and other reasons, the system incentives not publishing these, which is a pretty big problem.