[godelski]

A little of both. The problem is that evaluation of research work is insanely difficult. A lot of people think it's easy because "the world is objective, it either works or doesn't" but research is cutting edge and you're only chipping away at a much larger picture. It can take decades for a work to reveal itself as truly profound or utter shit. The problem, which I rant about in a longer comment, is that instead of acknowledging the noise we've embraced poor metrics and encouraged the hacking of those metrics. I call this Goodhart's Hell. People forget, metrics are models and all models are wrong. You have to constantly be questioning your metrics and determine how well aligned they are with your goals or else you'll drift (the environment moves, so your metric must move too).

I think actually the better way to solve this, which may seem paradoxical, is to actually increase funding. Not in size of single prizes for grants (well... we need that too, but that's another discussion), but in the availability. The reason being that the hacking is partially encouraged by the competition for a very scarce resource. A resource that compounds. Due to this (and some nuances, see other post) we're not actually rewarding those who perform the best work (we may actually be discouraging that) but those who become lucky. A "good work" is simply one with high citation counts, which is heavily weighted on the publicity around that work. Which is why top universities have big media departments, pay news publishers to advertise their works, and why survey papers generate huge counts.

The problem is that the system is rather complex and there are no simple or "obvious" solutions. "Good enough" is also not clear because too low order of an approximation can actually take you away from your intended goals, not a small step towards as one might think.

[blululu]

We have been increasing funding for 50 years and the problem has not been fixed. America currently sets the record in science funding (in raw amount and percentage of gdp) for any society in History and it seems to have these issues. Even if increasing funding would solve the problem, it is, at best, a short term fix. So long as there is growth in the system you can hide a lot of blight and rot. There is enough to go around so the zero sum bad actors don’t take the full pie. But the long problem with such a solution is that you cannot increase funding indefinitely. At somepoint there is a limit to how much money society can allocate to these projects. We need to make our spending work properly.

[caddemon]

Yeah any increase in funding needs to go to meaningfully different grant processes. Giving more money to the existing funding structures will not fix a damn thing. Because the present system is just woefully unscaleable. More money to current NIH -> more PhD students -> tenure track competition remains fucking ridiculous.

[Fomite]

We have been increasing funding in some ways. But, for example, the amount for a non-modular NIH R01 hasn't changed since the Clinton administration. A lot of individual grants are just as much work as they have been, more competitive, but pays for less science.

[WalterBright]

Any source of funding will spawn an industry designed around extracting every dollar from it.

[godelski]

Exactly. Or equally: money can only exchange hands by means of a leaky bucket. But I'd say that it's not a big problem that the bucket is leaky. Goodhart's Hell is when that extraction industry dominates or that bucket isn't so much leaky as it is missing the bottom which differentiates it from a tube. Some people call this peak capitalism and it's right to complain, but I think this happens in whatever system you use, just exhibits itself in whatever metric dominates (in our case capital/dollars. Also typically capital/dollars in communism too because both systems are explicitly about capital ¯\_(ツ)_/¯).