|
I have to strenuously disagree with you (respectfully, of course) on this issue, dang. My several years of participation in the University of Minnesota journal club on behavior genetics, in which we go over current papers, sometimes being amazed by the papers (especially the papers on the methodology of doing science right) and sometimes picking apart the papers line by line (especially the papers on social psychology), have taught me to the core that most scientists are doing a job narrowly conceived of as trying to get published. I think they mostly went into science because they also desire to seek truth and inform humanity, but institutional pressures sometimes keep them from achieving that ideal. A surprisingly huge percentage of working scientists, as John P.A. Ioannidis[1], Uri Simonsohn,[2] Peter Norvig,[3] and special issues of science journals[4] and methodological blogs[5] remind us, are actually working right at their own level of competence or even beyond their own level of competence in setting up experiments and observational studies to be validly generalizable. I do NOT assume that "that the people doing the work are idiots," but I do assume, based on huge numbers of historical and current events examples, that the people doing the work are fallible and subject to cognitive biases (as I am) and subject pressure to publish to stay funded. Moreover, announcements of results are subject to wishful thinking on the part of a university press office. Until a preliminary study has been replicated, I don't take it to be a statement of facts about the world. Even Wikipedia, which accepts some very dodgy user-submitted content, declares a content guideline that Wikipedia articles should be based on SECONDARY sources[6] (that is, sources by authors who have thought about and digested the primary research findings) rather by preliminary primary research findings. (Of course, for lack of enforcement, many Wikipedia articles break this rule.) It's especially important to establish high standards of sourcing for statements about human health and medicine and nutrition.[7] I genuinely think that many (too many) readers of Hacker News have no idea what an adequate sample size would be, for a given effect size, to validly infer from a preliminary study result a statement about the entire population. We should be talking about sample sizes all the time here (I agree, with more sophistication and nuance than we often do) as part of educating ourselves about basic science methodology in this community of intellectual discussion. That said, I heartily agree that "Betteridge's Law" is a useless Internet meme, even though it was popularized here by our esteemed site founder pg. We can do better, and we can raise the level of discussion here. I cherish the participants here who can speak knowledgeably about experiment design, about effect sizes, about observational studies as contrasted with experimental studies, and so on. I also delight when participants here share links to the prior scholarly literature, and especially when something is submitted here that is a better source than a press release.[8] Besides decrying crap, I like to applaud thoughtful discussion, so I regularly upvote comments that point us beyond the headlines to what issues researchers have to grapple with as they try to figure out the complexity of the world. [1] https://med.stanford.edu/profiles/john-ioannidis?tab=publica... http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/ [2] http://opim.wharton.upenn.edu/~uws/ [3] http://norvig.com/experiment-design.html [4] http://pps.sagepub.com/content/7/6/528.full [5] http://retractionwatch.com/ [6] https://en.wikipedia.org/wiki/Wikipedia:Identifying_reliable... [7] https://en.wikipedia.org/wiki/Wikipedia:Identifying_reliable...
[8] http://www.phdcomics.com/comics/archive.php?comicid=1174 |
Most published studies may well be false, but that's no basis for substantively discussing a specific one, any more than "most movies are bad" is a movie review.