[jonhendry]

Well, that's why the whole "replication" thing is important. One published result is interesting, but rarely definitive, and possibly wrong. (Or at least unusual for possibly difficult-to-determine reasons.)

This is another good reason to ignore the media hype for every new paper that comes out. (Besides the fact that journalists perform lossy compression on data.)

But it seems like it's how science is supposed to work: publish your results, see if others confirm your findings, because you might be wrong even if you seem to have done everything correctly and honestly to the best of your ability.

[St-Clock]

On the whole, I agree with you, but replication is often expensive and in some fields (e.g., software engineering research), it's not enough to warrant a publication in a high profile journal or conference (especially if you confirm the original findings). Heck, you may not even get funding to do that.

Regarding "if you seem to have done everything correctly", I don't think that any honest scientist can claim that his/her study had no limitation or flaw. I regularly review papers for big conferences and there is no such thing as a perfect paper/research project/study. It's more like a threshold: despite the issues, were the findings novel, relevant, and found through a rigorous process? Would the community learn anything valuable by reading this?

Articles like the one cited by OP are useful if they make scientists and normal "folks" realize the limitations of alpha values, they are useful if they make scientists reconsider some of their methods (and way of presenting findings), but they can be harmful if the readers conclude that most scientific findings are "false" and thus, that science is bogus because it cannot find the "truth". Science is rarely, if ever, about true and false, religion is.

P.S. I realize this answer was more about the article, and less about your reply, it's just that your reply prompted me to write something :-) Again, I agree with you!

[jonhendry]

"Regarding "if you seem to have done everything correctly", I don't think that any honest scientist can claim that his/her study had no limitation or flaw."

Right, that's what I was getting at. The scientist might believe they've done everything right, after checking and re-checking their work, but be missing some flaw or limitation in their work, their model, whatever.

I recall hearing once of an experiment that couldn't be reproduced, and it turned out to be due to some chemical property of the entirely normal laboratory glassware that one lab had used. Switching to another manufacturer removed the problem. (I'm probably messing up the details, like the consequences of the chemical properties of the glass. But the gist is correct. Different manufacturer of glassware cleared up a problem that was unexpected.)

[tokenadult]

Regarding "if you seem to have done everything correctly", I don't think that any honest scientist can claim that his/her study had no limitation or flaw.

Hear. Hear. This, of course, was the thrust of Richard Feynman's famous Caltech commencement speech on "Cargo Cult Science."

http://www.lhup.edu/~DSIMANEK/cargocul.htm (adapted HTML text)

http://calteches.library.caltech.edu/51/2/CargoCult.pdf (PDF version as published by Caltech)

As Feynman said, "The first principle is that you must not fool yourself--and you are the easiest person to fool. So you have to be very careful about that."