[noelsusman]

The alternative is consensus. As part of my research I found that a particular method of estimating disease prevalence in small geographic areas performed better than this other method when tested on real data from schools.

The tendency of the general public would be to look at that study and say the first method is better than the second method (assuming the general public would care at all, which they don't). That's incorrect. I would only conclude that method is actually better if several other people found similar results for similar methods in separate studies.

People tend to place far too much importance on one paper or one study. In theoretical research this can be okay sometimes, but in applied research this is almost always the wrong way to go.

[breischl]

Consensus seems to be the best available alternative, but it still doesn't seem to be very good in many cases. As the article pointed out it is not difficult to manufacture enough bullshit that it looks like a "competing consensus." Prime examples of that being the anti-vaccination movement, climate change deniers, and the pro-smoking studies from cigarette companies.

Counting up and evaluating all the studies is itself a time-consuming task, hence the existence of meta-analyses. But then you get into duelling meta-analyses, and you have to choose which one(s) to trust, and you're back to expert opinion and appeals to authority.

[astine]

"The alternative is consensus."

That's still an appeal to authority. It's just the authority of a group rather than an individual.

[effie]

I think he meant consistent experimental results by "consensus" there.