[Voultapher]

So many problems in science could be alleviated if we changed the culture and processes to multi replication. Your paper can only be accepted into a paper if an unrelated group can replicate the results with only the paper and associated submitted info.

[tnecniv]

This doesn’t scale. Sometimes only a small number of labs have the equipment needed to run an experiment, let alone the training on the equipment to do what needs to be done. This equipment can cost millions. Even if the results are correct, experimental science is very finicky by its very nature. Getting an experiment to work can take months.

[rtsil]

Fine, let's just do it on those that scale. Not all science is done in high-end, expensive labs, and this type of control will help the young researchers who are publishing their first works to gain more credibility and develop healthy habits.

Sadly, as long as the tyranny of publishing exists, the other researchers will always prefer working on their own things than replicating someone else's not yet published experiment.

[Voultapher]

I'd say if no one can afford to, or knows how to replicate the thing you did. What point is there in it? And only because such an approach has downsides it's not clear it means the overall benefit of lessening the replication crisis and forcing scientists to provide enough information their work can be replicated at all would be a huge win. Based on people like https://www.youtube.com/@AppliedScience experience, 9 out of 10 times there is some crucial information missing in the papers they are trying to replicate.

[anonymouskimmer]

> I'd say if no one can afford to

Because sometimes they can afford to build on your prior results, and advance the state of the art. Factoring in all of the malfeasance, what's the trade off between not publishing due to inability/unwillingness to replicate, and publishing bad results? We should know this tradeoff before significantly changing the status quo.

> or knows how to replicate the thing you did.

Agree completely.

[etrautmann]

Agreed - having done a PhD in an experimental neuroscience lab, there's a nonsignificant number of things that nobody else in the world, even in my own lab, can do. I can train the techniques to others, and do, but this is separate from scientific discovery. There are no incentives for someone else to spend their time replicating my work using unbelievably challenging and expensive methods. It just wouldn't work in practice without a more fundamental restructuring of the whole enterprise (which is also necessary but hard).

[somenameforme]

Like what? I can think of extremely few domains that any decent research university isn't well equipped to competently replicate. One of the very few benefits of the tuition explosion - even undergrads get relatively casual access to equipment worth millions of dollars.

Of course they can't replicate things like ultra high energy particle research, but these sort of obscure things make up a very negligible chunk of all science produced, even if it's quite an important little chunk.

[cycomanic]

I guess you're not thinking hard enough. I estimate >90% of research in most technical fields would require 10s of years to replicate unless you are one of the few labs already working on the same topic.

For example in on of the research areas I'm familiar with (optical communications), there are maybe 10 academic labs in Europe (and even less in the US) who have the equipment to reproduce some of our experiments. In our lab there is 1 PhD student who could pull off reproducing the more sophisticated experiments (because he is the one focusing on communications) it took him 2 years to get to that stage.

This is an relatively easy area, i.e. equipment is largely off the shelve, very applied with lots of industry involvement. There are plenty of experiments published which could only be done in 2 labs (both of them industrial), just due to the cost of the required equipment.

In other areas (e.g. with fabrication in the clean room) reproduction would require even more time investment.

Don't get me wrong, reproducing results is important, but what people don't realise it happens all the time when people do adopt part of published results into their research. Mandatory reproducing results would just create large overheads which would get us nowhere.

[somenameforme]

It's entirely possible there are fields with needs I am unaware of or not considering, but your response is not compelling and sounds like hand-waiving. Exactly what equipment are you talking about?

I suspect you are likely grossly underestimating the available supplies at many research university in the US. For instance things like class 100 clean rooms are basic facilities. Many (and I want to say most) research universities also have partnerships with (if not ownership) of various specialized labs in the surrounding areas for more specific purposes. For instance the NASA Jet Propulsion Lab is managed by Caltech.

[cycomanic]

Regarding equipment:

Realtime oscilloscope at least 4 channels > 50 GHz bandwidth $0.5M (for some research you need >=12 channels so multiply that number)

Arbitrary waveform generator 4 channels > 45 GHz bandwidth $0.3M (again you might need more than for channels)

RF amplifiers, electro-optic components etc. easily cost $2000-$5000 each and you need several (4-8 at least) of these. The RF cables and connectors/adapters easily cost several thousand $ each.

Fibre components, subsystem components (e.g. a WSS of which you will likely need 4 or so is $50k).

And I will certainly not let a student without training touch the sensitive high-speed RF equipment.

Regarding your comment on clean-room. For many fabrication purposes class 100 is not sufficient (also calling it basic facility is quite rich). And the equipment in is very expensive, LPCVD machines, E-beam, other lithography ... is $10sM. Most universities I'm aware of require fees (typically paid from grants) of $10sk per year to use the facilities (those are the reduced rates for university staff). The training/certification on the equipment typically takes about 1year.

Regarding JPL, yes it's managed by Caltech, what do you think NASA will say if Caltech professors will ask for a student to use the facilities to verify some paper? Sure, lets delay the next Mars mission a year or so, to let some PhD students try stuff in the labs.

I think you seriously underestimate what the cost of using all that equipment is and how much training is involved to be allowed to use it. You definitely don't want any

[somenameforme]

When I say a class 100 cleanroom is a 'basic' facility, I mean it's one you'll find at any decent research university in the US, and it is. If there's something that can be reasonably expected to be required for cutting edge research, you'll find it. As for lab access, my experience is in CS. I was granted access to a globally ranked supercomputing system paired alongside a paired largescale audio-visual facility. The only requirements for access were to be whitelisted (involved in research) and registered. After that of course you needed to slot/reserve time, but it was otherwise freely and unconditionally available.

It's difficult to really explain how much money is spent in top US universities. It's as if there's a fear that revenues might manage to exceed costs. But one of the practical benefits of this is that bleeding edge hardware and supplies, at costs far greater than anything you've listed, is widely and readily available.

[rleigh]

I think the cost and complexity of reproducing work is somewhat overestimated, as is the specific expertise of individual researchers, though maybe your field is exceptional in this regard.

Primary research, pioneering new techniques and equipment to explore the unknown, is time-consuming and costly and requires a lot of original thought and repeated failure until success is achieved. However, reproducing that work doesn't involve much of this. It's taking the developed methodology and repeating the original work. That may well involve expensive equipment and materials, and developing the technical expertise to use them, but that does not involve doing everything from scratch and should not take anything like as long or cost as much.

I also believe that we far too readily overestimate the specific special skills which PhD students and postdoctoral researchers possess. Their knowledge and skills could likely be transferred to others in fairly short order. This is done in industry routinely. A PhD student is learning to research from scratch; very little of their expertise will actually be unique, and the small bit that is unique is unlikely to be difficult for others to pick up. I know we don't like to think of researchers as replaceable cogs, but for the most part they are.

My background is life sciences, and some papers comprise years of work, particularly those involving clinical studies. However, the vast majority of research techniques are shared between labs, and most analytical equipment is off the shelf from vendors, even the very expensive stuff. Custom fabrication is common--we had our own workshop for custom mechanical and electronic parts--but most of that could have been handled by any contract fabricator given the drawings. And the really expensive equipment is often a shared departmental or institutional resource. Most of the work undertaken by most of the biological and medical research labs worldwide could be easily replicated by one of the others given the resources.

Depending upon the specific field, there are contract research organisations worldwide which could pick up a lot of this type of work. For life sciences, there are hundreds of CROs which could do this.

As one small bit of perspective. In my lab a PhD student worked on a problem (without success) for over a year. We gave it to a CRO and they had it done in a week. For less than £1000. The world is full of specialists who are extremely competent at doing work for other people, and they are often far more technically competent and efficient than academic researchers.

[doctorpangloss]

It is a misconception that replication doesn’t occur. In vanguard basic science, every journey crosses replicating the most recent results.

[Voultapher]

I'd like to learn more, could you please provide some sources.

Based on the recent Ranga Dias fraud, I'd think replication culture still has a long way to go.

[coolhand2120]

Is that true? Do you have any citation? It sounds incredible. Every study?

[pcrh]

It's broadly true that most research is eventually subject to attempts at replication. However the replication isn't explicitly attempting to reproduce the exact prior research, but to build on it.

If the foundation established by the prior research is flawed, attempts to build on it will usually fail.

[ska]

You don't really need that (and it would be too expensive), you just need to fix the incentives so that people do more replication.

[ffgjgf1]

Someone will need to pay that other group though. Sometimes quite a lot.

[Voultapher]

I'd argue it's worth it.

Some practical ideas, when allocating the money for research 30% or so has to be kept back for replication. Public science is mostly government funded. And the government could allocate X% percent of the budget for replication work. It sounds like a solvable problem to me.

[ffgjgf1]

I agree in principle but forcing institutions or governments to allocate money for this and enforce is pretty much an unsolvable problem at this point IMO.

[Voultapher]

Yeah, as I said it's also a question of culture and that's very difficult to change.

[hcks]

If this had been enacted, there would never have been any scientific progress. Should tell you something

[Voultapher]

You base this claim on what exactly?