[marcus_holmes]

So, what's the best proposal or line of thinking coming from all this deep thought about how to solve it?

Because you can't possibly be saying that the current system is the best we can do, or that the problem is intractable.

(I agree with most of your proposals about software engineering btw. We should do more of this).

[adminprof]

It'd be a process: participant in a community in a subfield, listen to ideas that have been tried and outcomes, come up with an idea using those lessons learned and success metrics, build consensus around a new idea, test in a single-instance (like one conference, grant review panel, tenure committee at one university), share lessons to the field, do this for a couple of years to show clear success, expand to multiple events in the field, become an exemplar field for that idea and "infect" other fields

Sounds slow but there's thousands of such experiments happening simultaneously right now. This is how a long of major field-sized changes have happened, like the transition to conferences from journals (which had many initial problems like during tenure review or a lack of quality in reviews), etc. Ideas will lose traction at various stages (for example, there was a movement some time ago to use alpha=0.001 instead of alpha=0.05 for null hypothesis testing, which has been limited to that field or subfield).

[marcus_holmes]

Interesting, thanks.

Is the move to pre-publishing servers (like arxiv) a part of this? How does SciHub (and similar) figure into it?

It does sound slow, and a bit trivial, if I'm honest. Are there any examples you can share of successful experiments that have travelled across field boundaries?