|
On the whole, I agree with you, but replication is often expensive and in some fields (e.g., software engineering research), it's not enough to warrant a publication in a high profile journal or conference (especially if you confirm the original findings). Heck, you may not even get funding to do that. Regarding "if you seem to have done everything correctly", I don't think that any honest scientist can claim that his/her study had no limitation or flaw. I regularly review papers for big conferences and there is no such thing as a perfect paper/research project/study. It's more like a threshold: despite the issues, were the findings novel, relevant, and found through a rigorous process? Would the community learn anything valuable by reading this? Articles like the one cited by OP are useful if they make scientists and normal "folks" realize the limitations of alpha values, they are useful if they make scientists reconsider some of their methods (and way of presenting findings), but they can be harmful if the readers conclude that most scientific findings are "false" and thus, that science is bogus because it cannot find the "truth". Science is rarely, if ever, about true and false, religion is. P.S.
I realize this answer was more about the article, and less about your reply, it's just that your reply prompted me to write something :-) Again, I agree with you! |
Right, that's what I was getting at. The scientist might believe they've done everything right, after checking and re-checking their work, but be missing some flaw or limitation in their work, their model, whatever.
I recall hearing once of an experiment that couldn't be reproduced, and it turned out to be due to some chemical property of the entirely normal laboratory glassware that one lab had used. Switching to another manufacturer removed the problem. (I'm probably messing up the details, like the consequences of the chemical properties of the glass. But the gist is correct. Different manufacturer of glassware cleared up a problem that was unexpected.)