[chasing]

Look. Clearly science makes progress that's "true" in the sense that it becomes useful and can be used as functional models of the way things work.

This video's using a very reductionist kind of statistics to point out that, yes, an individual piece of research making a claim might have a good chance of being wrong. Which is why science doesn't say, "Oh, well Larry just proved that Saturn orbits Uranus so let's just never think about that again and instead move on to proving that the Sun is fully powered by excess heat radiating off of Elon Musk's brain." Science is a process that works in aggregate, using a large volume of research and scientists checking one another as a way to smooth over this very imperfect process. Over time. Science checks itself. That's the whole point. That's why it reaches some pretty damned good conclusions about the way things work.

So.

I don't know what the point of this video is. Science is wrong? Scientists are stupid? The Economist is smart? I should believe the Republicans when they say the Earth couldn't possibly be warming because that one time it snowed in a part of Texas where it never really snows all that often?

[pygy_]

> Science checks itself.

Not that much, actually. There's no point in reproducing the work of others to advance your own scientific career. Unless your research directly directly depends on them, it is counterproductive.

[epistasis]

In the important areas, it does check itself. If a scientist has been doing some interesting work, she will soon find herself with colleagues working on the same area. If she has not been doing something interesting, that line of research does not get cited, dies out, and does not get funding.

In active learning, there's a tradeoff between exploration and exploitation. Science does this too, but heuristically.

[capnrefsmmat]

Not necessarily true. Ioannidis did a review of 49 studies cited more than 1,000 times in the medical literature -- prime candidates for being replicated or tested by future results. Of those,

- 16% were found by later studies to be wrong

- 16% were found to have exaggerated the size of effects they claimed to detect

- 44% were replicated, and

- 24% were unchallenged and unreplicated by later literature.

So a full quarter of incredibly prominent research was never tested.

Ioannidis, J. P. A. (2005). Contradicted and initially stronger effects in highly cited clinical research. JAMA, 294(2), 218–228. doi:10.1001/jama.294.2.218

[epistasis]

Thank you for a very informative and helpful reply! However, I would like to see this work validated before I fully accept his conclusions.

Regarding the 24%, if you look in Table 2 of that study, of the 12 that were "unchallenged and unreplicated", the notes for 10 list follow on studies that would reveal problems with the initial study, were the initial study to be "wrong."

That leaves 4% that are "unchallenged and unreplicated" yet have 1000 citations. So I would be quite interested to know the take of the researchers in the field on those papers, and if they would agree with Ioannidis' assessment on those 2 papers.

[001sky]

If she has not been doing something interesting, that line of research does not get cited, dies out, and does not get funding.

Could you explain how this verifies results? Citation (alone) isn't verification. And its safe to say followers and funding correlate with one another, at least statistically.

[bloaf]

Say someone publishes a paper that has the conclusion: If you preform procedure X, you get carbon nanotubes with property Y.

Lets then go on to say that someone publishes a paper which states: carbon nanotubes with property Y will theoretically allow us to do Z.

If Z is something interesting and worthwhile, it won't be long before people read both papers and decide to try using procedure X to make Y nanotubes and then measure their ability to do Z. If no one can actually get to Z, then one or both of those papers was wrong, and the field will recognize that. The papers may not be formally retracted, but everyone will move on and stop referencing the incorrect papers.

[epistasis]

You're reversing what I'm trying to say, I'm saying that verification or invalidation causes citations to occur, not that citation is verification. And that work that doesn't get validated dies out. Even work that gets "invalidated" in some sense can reveal other truths, or necessary changes in paradigms.

This is why scientists check the citations of a paper when considering its contents. It's important to ask if other people followed up on this, and what have they found if they did follow up? New papers, without any citations, must be held in a state of meta-information, until there's follow up papers. Old papers with few citations, and no validation citations, must also be considered as in a state of meta-information. Sometimes, really important things like Mendel's genetics get lost for decades in this state, until they are rediscovered, but it's fairly rare.

[pygy_]

Work often gets cited without being either replicated or invalidated.

Do you do any kind of research? You seem to have an unrealistically rosy perception of the scientific process as it happens concretly.

[epistasis]

>Work often gets cited without being either replicated or invalidated.

I'm not sure why you're bringing up this fact, which is not at all inconsistent with my comment. What's your implication?

>Do you do any kind of research? You seem to have an unrealistically rosy perception of the scientific process as it happens concretely.

In the past decade, I've spent probably 70% of my time on scientific research. At this moment I'm procrastinating from writing a letter of support on a grant. What, in particular, do you think I'm unrealistically rosy about?

[gabriel34]

Actually there are loads of scientific papers only reviewing other papers (usually a set of them), trying to infer conclusions and criticizing them on methodology and rigidity. Scientific careers advance both in quality and in quantity of papers published. Not everyone gets featured on Science or Nature.

[tga_d]

This is actually what I'm doing for my undergrad senior thesis right now. It's in theory-heavy CS, but there are some results that are based on some moderately difficult math, and experimental evidence of those results, which I'm independently verifying. So, it seems to apply. It's not the most exciting thing to be doing in the field, but it's more interesting than most of my other classes, and can be used for later work.

[dhaivatpandya]

Agreed. Try getting funding for verifying a paper's results. If you can't publish results, it'll be a hard time convincing anyone that your work is worthwhile.