[davidktr]

>Science, since time immemorial, has relied on the systemic replication of any presented result or finding. Reproducing experiments and their reported results remains a cornerstone of the validation of any scientific theory.

No, no it really hasn't. It has relied on the ability to make predictions based on pusblished theories, methods, laws etc. Even for hard-science experiments it's not even clear how you could record all the required knowledge to replicate an experiment. Every configuration, every machine, every particle in the air, every bit of software.

I really wish people engaged more with the actual history of science instead of what they believe it to be.

edit. To give a little more meat to my rant here's a good reading (https://plato.stanford.edu/entries/scientific-reproducibilit...):

"If replication played such an essential or distinguishing role in science, we might expect it to be a prominent theme in the history of science. Steinle (2016) (...) claims that the role and value of replication in experimental replication is 'much more complex than easy textbook accounts make us believe' (2016: 60), particularly since each scientific inquiry is always tied to a variety of contextual considerations that can affect the importance of replication. Such considerations include the relationship between experimental results and the background of accepted theory at the time, the practical and resource constraints on pursuing replication and the perceived credibility of the researchers. These contextual factors, he claims, mean that replication was a key or even overriding determinant of acceptance of research claims in some cases, but not in others."

The history of replications is extremely nuanced. Empirical results and by extension replications are one line of argument in scientific discourse, but by no means the only one.

I personally hold that valid predictions in the context of interesting problems are where it's really at. In the "Structure of Scientific Revolutions", Kuhn argues that at some point paradigms cannot make THESE kind of predictions anymore. Revolutions do not happen because of failed or missing replications.

Therefore, stating science "has relied on replication" is historically and epistemologically false. It's also misleading because the replication crisis happens due to a lack of theory and misguided incentives, not because some discipline has left the holy path of finding truth.

[dpleban]

> It has relied on the ability to make predictions based on pusblished theories, methods, laws etc

This might be a semantic argument, but what you describe is replication. Imagine every scientist would say "In my experiment, I perfectly predicted this and that, oh but no one else would ever be able to run that experiment again, so just trust me, ok?"

Replication/reproducibility isn't about logging every configuration, machine or particle. It's about being able to run the same method and get the same result. If that isn't the case, how do we know the predictions are correct?

[ookdatnog]

> Imagine every scientist would say "In my experiment, I perfectly predicted this and that, oh but no one else would ever be able to run that experiment again, so just trust me, ok?"

I don't know much about this topic, but isn't this kind of what's happening with for instance the Large Hadron Collider? We seem to collectively trust the results from LHC experiments, even though no one can replicate them because there is only one LHC and AFAIK no one is building another particle accelerator for similar or greater energy levels. There's no guarantee that we'll see another particle accelerator of that scale in our lifetime.

It seems to me that your point implies that LHC experiments shouldn't be considered to be science, since they cannot practically be replicated any time soon (except at the LHC itself, which somewhat defeats the purpose of replication). But I (not a physicist) find myself quite trusting of their results tbh. I'm not sure I can fully articulate why, and I'm not sure I have a good reason for it.

[ziddoap]

>Science, since time immemorial, has relied on the systemic replication of any presented result or finding

>>No, no it really hasn't.

What do you mean no it hasn't? Reproducibility of scientific findings is certainly a cornerstone of science, at least according to all definitions of "scientific method" I have ever found.

>Crucially, experimental and theoretical results __must be reproduced__ by others within the scientific community.

https://en.wikipedia.org/wiki/Scientific_method

>Reproducibility, also known as replicability and repeatability, __is a major principle underpinning the scientific method.__

https://en.wikipedia.org/wiki/Reproducibility

[dekhn]

At the state of the art, reproducibility is less important for competitors than simply knowing that your competitor- who you trust is nearly as competent as you- did achieve what they claim.

I see reproducibility as an aspirational goal, at least in the subset of "discovery" science where the competition is to be first to identify a new scientific principle. Reproducibility is more important in another area- people building tools for others to use for their own research.

[davidktr]

see my edit

[ziddoap]

>Empirical results and by extension replications are one line of argument in scientific discourse, but by no means the only one.

Someone said science relies on reproducibility and you outright dismissed that with a "no, no...", now it sounds like you're saying "well, yes but it's nuanced".

If you just said "yes, but it's nuanced" from the start without being so dismissive you probably wouldn't have been met with disagreement.

[davidktr]

The full quote in question: "Science, since time immemorial, has relied on the systemic replication of any presented result or finding."

This is wrong, plain and simple. It paints a picture of necessary and sufficient conditions for scientific progress which are incorrect.

- The vast majority of "results and findings" is not looked at anybody other than the researchers directly involved. If you have 5 people on a paper, be sure at at best 3 of them have seen actual data, or were even involved in the experiment.

- Where is the systemic replication, exactly? Replication rates vary considerably across fields. And, of course, only selected results are replicated.

- If there was a systemic replication of "any result and finding", how is it possible that there is a replication crisis at all? Should the bad apples not have been found long ago?

If science would, in fact, rely on such a system, doing replications would be a normal part of everyday scientific work. It is not, not by a long shot.

So you can conclude that either science is not happening at all (not sure when it did though), or that the quoted premise is incorrect.

[leecarraher]

computer science imo somewhat gets the highest level of scrutiny when it comes to reproducibility simply because so many people can, in theory, reproduce an experiment without any special expertise. this cannot be said for many other sciences. Others require costly equipment, expertise and time, and even then, may fail to correctly reproduce an experiment. Imagine trying to reproduce a CERN finding in your backyard, or a James Webb observation, a social science survey, or medical study. DL isn't bullsh-T it's one of the few democratized enough sciences (similar to math) that has a reasonably low barrier to entry. unfortunately that may be its undoing, because it means the masses want the off the shelf docker container that just does the toy thing the experimenters spent potentially years working out. Could documentation be better? of course, but before declaring a field BS, maybe compare it to some other scientific fields.

[dchftcs]

Even then the predictions need to be confirmed and proven not to be a fluke.

While technical reproducibility isn't always required (in case of observing a rare event, and there is enough expertise to evaluate the fidelity of the experiment / observation), it's also a bit of a strawman to attack this point specifically, because in any case science advancement needs a body of evidence appropriate for the theory being tested, and replicability is crucial for a field like DL where it should have been relatively easy, and where the basic premise axiomatically requires reproducibility (that a reapplication of the same techniques should yield comparable results).

[unixhero]

He refers to scientific positivism.

https://plato.stanford.edu/entries/logical-empiricism/

[bejelentkezni]

The most basic and implicit prediction you can make is "if I do this again I'll get the same result."