I feel like there's some fundamental fallacy in the idea that "a declining rate of scientific advancement" is a sign that the field is somehow being corrupted or rotting out from the inside.
Science isn't like other commodities. In most of recorded history it is only ever produced, never destroyed [1], and the product is basically free to replicate [2]. The result is massive inflation: it might be hard to make a profit growing corn the same way we did 200 years ago, but doing a 200 year old science experiment is utterly pointless outside a classroom demonstration.
So making science that is worth paying for is just always going to get harder. And yet we equate science with other industries when we expect anything less than billion dollar experiments to yield fundamentally interesting results. This doesn't mean science is somehow getting worse, or that the practitioners are to blame, it just means it's evolving to attack much more difficult problems.
All this being said, there are plenty of ways to reform to keep the progress going: reproducibility is theoretically easier than ever, and yet many journals aren't requiring open datasets or public code. We need to keep the pressure on to evolve in a positive way, not just throw up our hands because things are harder than they were when we knew less.
[1]: Ok, there are some examples were lots of information was destroyed, and a bias from what is recorded.
[2]: I don't mean repeating the same experiment, just that the results from one experiment are trivially disseminated to millions of people.
Could we blame the "industrialisation" of PhDs on that we should expect less impact from each researcher and thus the obvious policy the keep interesting research happen is more researchers?
Pretty influential one: https://www.aeaweb.org/articles?id=10.1257/aer.20180338 "The number of researchers required today to achieve the famous doubling of computer chip density is more than 18 times larger than the number required in the early 1970s. More generally, everywhere we look we find that ideas, and the exponential growth they imply, are getting harder to find."
That one, though, is because we are running into physical limits: if we want to build things out of atoms, we can't make features that are half an atom thick. Even above that scale, physical effects that used to be ignorable, like quantum tunneling, no longer are.
From the late 70s through about 2005, scaling semiconductor generations was easy. MOSFET scaling followed rules formulated by Dennard, which provided a fairly easy method of scaling semiconductor designs from one generation to the next, keeping power density roughly constant and continually improving performance. The problem is that by around 2005, if you did it that way, your gates were no longer switches, they were dimmers, and leakage power started to dominate, and that meant that chip architectures had to change radically to keep on scaling.
So, we can no longer just scale designs from one generation to the next, we have to come up with completely new approaches. That's much harder.
You could argue that those current researchers are doing a lot more than those in the 70s. It is difficult to quantify how much harder the doubling problem becomes every time, and how much more effort it takes to solve. But the fact that, after decades of yearly exponential improvement, costs have consistently grown only linearly. More specifically, only x18 cost for doubling after roughly 40 iterations (2^40 is massive). I mean that’s phenomenal by any standards.
Science isn't like other commodities. In most of recorded history it is only ever produced, never destroyed [1], and the product is basically free to replicate [2]. The result is massive inflation: it might be hard to make a profit growing corn the same way we did 200 years ago, but doing a 200 year old science experiment is utterly pointless outside a classroom demonstration.
So making science that is worth paying for is just always going to get harder. And yet we equate science with other industries when we expect anything less than billion dollar experiments to yield fundamentally interesting results. This doesn't mean science is somehow getting worse, or that the practitioners are to blame, it just means it's evolving to attack much more difficult problems.
All this being said, there are plenty of ways to reform to keep the progress going: reproducibility is theoretically easier than ever, and yet many journals aren't requiring open datasets or public code. We need to keep the pressure on to evolve in a positive way, not just throw up our hands because things are harder than they were when we knew less.
[1]: Ok, there are some examples were lots of information was destroyed, and a bias from what is recorded.
[2]: I don't mean repeating the same experiment, just that the results from one experiment are trivially disseminated to millions of people.