[bumby]

I think that's reasonable for the "institutional review" aspect of the OP. But regarding the "peer review" aspect, I don't think it works, at least in the bulk of the current framework. Peer-review is typically performed by separate organizations, independent of the funding organization. To expect a national-level organization to essentially take over the duties of peer-review journals is a very big ask (not in small part because the current system benefits from free labor from the reviewers).

[codekilla]

> To expect a national-level organization to essentially take over the duties of peer-review journals is a very big ask

Fair, but what is the alternative that would actually work? What is the budget of all of the journals compared to the NSF+NIH? Is medical research that is true, and certainly actionable worth as much as an F whatever fighter jet? People will have to decide.

[bumby]

The tradeoff is a different arguement and a digression, so while it's likely we agree, I'll side-step it here. I do think it's an uphill battle to pursue a massive reappropriation of funds though.

There are some alternatives that I'm aware of. Here's a few:

1) One is to allow journals to focus on less-than-great results. Right now the focus is on novelty, so there is an incentive to show that your work has some new, great outcome. But there's also value in showing "Hey, we thought this idea had legs but it turns out it didn't." Publishing that work should be part of science but right now its not. (As a side benefit, you could prevent a lot of researchers wasting effort on the same idea simply because they weren't aware that other people already tried, and failed.)

2) Journals can put a premium on sharing your data and code during the review process. Right now, it's often just up to the author and there are lots of veils to hide behind that essentially give the impression of sharing data, but not in a very useful way.

3) Give value to replicating work. Maybe not as much prestige as creating new work, but showing that it can be replicated obviously has value to society as a whole. Most of the time this won't get published, except in the cases where it's sensationalized, like fraud. (This effect is related to #1)

4) Journals can do a better job vetting their reviewers. They struggle to get timely reviews and reach to anyone who accepts the duty. Reviewers may agree to review something they have little background in, and as a result, it's easier to skirt bad articles through the system.

[codekilla]

I don’t disagree with any of these points, it’s just I’ve been involved in open science circles, where these things are always mentioned, and I just don’t see any material progress (maybe I’m not looking that closely though). I think the reason for the lack of progress is mainly funding—so until someone gets serious about funding (billionaires or taxpayers), it just seems like the same merry go around. It’s very expensive to replicate biomedical studies—but it’s the only thing that works. Maybe the tide is turning though and simply incentivizing/protecting grad students to become whistleblowers will do more good, but I fear this case was more the exception than the rule.

[bumby]

I think the difference is between a ground-up or top-down approach. Maybe both are needed. My current stance is that while a top-down approach would work, there's very little chance of it happening. For one, government research funds have largely flat-lined in the last 20 years, and expecting them to take on more costs for managing peer review would likely exacerbate the problem. I also don't see the govt clamoring for additional administrative burden. I don't think replication has to be the only method (although I think it's probably the best). Opening the data to the public can do a lot to suss out bad practices or outright fraud, as we saw with the Ariely situation. The progress has been slow, for sure, but I think there is some. For example, there are now journals that specialize in publishing "non-surprising" results.