You just described the majority of scientific papers. A "working set of instructions" is not really feasible in most cases. You can't include every piece of hard- and software required to replicate your own setup.
Then don't call it science, since it doesn't contribute anything to the body of human knowledge.
I think it's fascinating that we can at the same time hold things like "one is none" to be true, or that you should write tests first, but with science we already got so used to a lack of discipline that we just declare it fine.
It's not hard to not climb a tower you can't get down from. It's the default, actually. You start with something small where you can describe everything that goes into replicating it. Then you replicate it yourself, based on your own instructions. Before that, you don't bother anyone else with it. Once that is done, and others can replicate as well, it "actually exists".
And if that means the majority of stuff has to be thrown out, I'd suggest doing that sooner rather than later, instead of just accumulating scientific debt.
This is a very simplistic view. Why do believe QC departments exist? Even in an industrial setting, companies make the same thing at the same place on the same equipment after sometimes years of process optimisation of well understood technology. This is essentially a best case scenario and still results fail to reproduce. How are scientists who work at the cutting edge of technology with much smaller budgets supposed to give instructions that can be easily reproduced on first go? Moreover how are they supposed to easily reproduce other results?
That is not to say that scientist should not document the process to their best ability so it can be reproduced in principle. I'm just arguing that it is impossible to easily reproduce other people's results. Again when chemical/manufacturing companies open another location they often spend months to years to make the process work in the new factory.
> companies make the same thing at the same place on the same equipment after sometimes years of process optimisation of well understood technology. This is essentially a best case scenario and still results fail to reproduce.
We're not talking about 1 of 10 reproduction attempts failing, we're talking about 100%. And no, companies don't time and time again try to reproduce something that has never been reproduced and fail, to then try again, endlessly. That's just not a thing.
> it is impossible to easily reproduce other people's results
We're also not talking about "easily" reproducing something, but at all. And in principle doesn't cut it, it needs to be reproduced in practice.
Imagine two scientists, Bob and Alice. Bob has spent the last 5 years examining a theory thoroughly. Now he can explain down to the last detail why the theory does not hold water, and why generations of researchers have been wrong about the issue. Unfortunately, he cannot offer an alternative, and nobody else can follow his long winded arguments anyway.
Meanwhile, Alice has spent the last 5 years making the best possible use of the flawed theory, and published a lot of original research. Sure, many of her publications are rubbish, but a few contain interesting results. Contrary to Bob, Alice can show actual results and has publications.
Who do you believe will remain in academia? And, according to public perception, will seem more like an actual scientist?
Academic science isn’t just the doing science part but the articulation and presentation of your work to the broader community. If Bob knows this space so well, he should be able to clearly communicate the issue and, ideally, present an easily understandable counter example to the existing theory.
Technical folks undervalue presentation when writing articles and presenting at conferences. The burden of proof is on the presenter, and, unless there’s some incredible demonstration at the end, most researchers won’t have the time or attention to slog through your mess of a paper to decipher it. There’s only so much time in the day and too many papers to read.
In my experience, the best researchers are also the best presenters. I’ve been to great talks out of my domain that I left feeling like I understood the importance of their work despite not understanding the details. I’ve also seen many talks in my field that I thought were awful because the presentation was convoluted or they didn’t motivate the importance of their problem / why their work addressed it
I disagree that Bob doesn't produce actual results, or that something that is mostly rubbish, but partly "interesting" is an actual result. We know the current incentives are all sorts of broken, across the board. Goodhart's law and all that. To me the question isn't who remains in academia given the current broken model, but who would remain in academia in one that isn't as broken.
To put a point on it, if public distrust of science becomes big enough, it all can go away before you can say "cultural revolution" or "fascist strongman". Then there'd be no more academia, and its shell would be inhabited by party members, so to speak. I'd gladly sacrifice the ability of Alice and others like her to live off producing "mostly rubbish" to at least have a chance to save science itself.
I think it's fascinating that we can at the same time hold things like "one is none" to be true, or that you should write tests first, but with science we already got so used to a lack of discipline that we just declare it fine.
It's not hard to not climb a tower you can't get down from. It's the default, actually. You start with something small where you can describe everything that goes into replicating it. Then you replicate it yourself, based on your own instructions. Before that, you don't bother anyone else with it. Once that is done, and others can replicate as well, it "actually exists".
And if that means the majority of stuff has to be thrown out, I'd suggest doing that sooner rather than later, instead of just accumulating scientific debt.