[javert]

> You're absolutely on the ball, except that everything is ~100x worse and more hopeless than you're describing :( Far from overstating it, you're understating it.

I mean, you can't get 100x worse than "60% of research is invalid."

> Absolutely

If you get enough pressure on Congress, you can always make changes. I wish you wouldn't be so belligerent.

Society does change, but slowly. You would probably have said slaves will never be free, or women will never vote, or racial integration of schools will never happen.

> Gates doesn't care. Like all the other sources of funding for public sector research, he wants to be a philanthropist cheerleader for Science™.

You don't know that. Unless you've read his secret diary, or something. I suspect he does care. And I think the evidence weighs more heavily on my side of the argument. But you aren't debating the evidence; you're just claiming to know, which you obviously don't.

[mike_hearn]

I'm not saying fixes will never happen - that's a strawman - just pointing out the immense scale of the challenges if you try and do it via moral suasion. It originates (IMO) in the corruption created when funders of work don't actually care about the results, just being seen to fund it i.e. governments, foundations. As such there are very few pressure points that can work because the system is so well insulated. Congress did make a small effort in the past on this (more than most countries did), back in the 70s. That's the origin of the OSI. However the effort flamed out shortly after and the OSI has been largely defunct since then, though it still exists and consumes budget.

To make progress here probably need political parties that take up academic reform (or defunding) as a voter wedge issue, and campaign on it for many years. Based on the failure of initiatives that tried to solve these issues so far, I'm very skeptical of any effort that isn't based on some sort of populism. Realistically this means any attempt to dramatically improve standards is going to get tangled up in culture war, as after all, you can't divorce an attempt to raise scientific standards from specific fields and claims (if you do it's admitting that they should be ignored). To get political momentum you need to be able to point to examples of claims that are wrong, and they have to be claims people actually care about.

Re: 60%, that's your personal estimate, right? My impression is that it varies a lot by field. In some fields you don't really get much invalid research at all, it's a curiosity. In other fields 100% of papers are useless because the underlying premises of the field are themselves wrong.

"You don't know that. Unless you've read his secret diary, or something."

I haven't read his secret diary, no. I have talked to someone who worked in epidemiology on malaria research, who described to me how the field is totally distorted by Gates Foundation funding, I've read many of his statements throughout COVID and read a skeptical review of his non-secret book (if you can get past the invective at the start the review is pretty decent):

https://www.eugyppius.com/p/we-must-find-a-way-to-prevent-bi...

... which reinforced the overall impression: Gates is a cheerleader. His approach to find the hierarchically most important people, ask them what they think and then repeat it uncritically, whilst distributing grants to more or less anyone who says they'll make Gates's personal goals come true.

It's also the case that if Gates was reading the outputs of his funded researchers and is as smart as usually claimed, he would have long ago noticed the problems. Yet his book boils down to: what we need next time is way more of all that. He is aware lots of people think Ferguson is a fraud, but thinks that's only because they were misinformed by the press. As far as Gates is concerned Ferguson is great and he repeats Ferguson's defences of his own work verbatim, even though they aren't accurate. As the reviewer points out, it's impossible to believe given what's written in his book that he ever actually read Ferguson's research (I have read it, and his model code, very carefully). Gates' one concession is that the vaccines weren't the silver bullets they were promised to be, but as for everything else - well, he acts as if he's completely unaware that any problems exist.

Now, as you observe, this might be an act. Nobody wants to spend decades lavishly funding people to engage on a noble mission and then one day admit, actually, they were mostly scamming me and we didn't get much out of it. The loss of face would be impossible to handle. Even if Gates did know, we might expect him to act as if he didn't. Still we'd hope he'd find subtle ways to improve things without outright admitting to the problem in plain language. I've never seen any evidence of this.

At least with Gates there's the theoretical possibility he could have a Damascene Conversion and start enforcing rigorous standards. With governments it really does need to become a political issue before anything can happen, as every time people try and improve standards via government, or purely internally inside academia, the new rules seem to be immediately subverted and everyone carries on as before.

[javert]

I appreciate your thoughts.

I agree that pushing for cultural/political change is a hard way to go given the populism in America. I'm an American. On a personal level, I think the country is so broken that it's not worth fixing, and would like to expatriate.

> Re: 60%, that's your personal estimate, right? My impression is that it varies a lot by field. In some fields you don't really get much invalid research at all, it's a curiosity. In other fields 100% of papers are useless because the underlying premises of the field are themselves wrong.

Yes, it was my estimate. But it's just a number I threw out there. I don't disagree with what you're saying here, but I'm more focused on the structural problems that exist to some (varying) degree in every field. For example, in every field, funding is corrupted. So I wouldn't say there are some fields that don't get much invalid research at all. Outright fraud is not the only kind of "invalid" research. It's still "invalid" if it's honest research but exploring the wrong path because there's some old fart at the funding agency who has a lot of friends who are exploring that particular path, which actually has been played out for 10 years already.

> Blog post about Bill Gates

I appreciate your sharing this. I think it's easy to construct this sort of narrative. That doesn't mean it's true. It might be. I don't know enough to tell.

I really think someone like Bill Gates could see the kind of research validation institute I'm proposing as a "win." I'm not saying all the research he's funding is worthless. I don't believe that, personally. I'm just saying, we can boost the quality and effectiveness of research if we have this kind of institute. Maybe our research process as a civilization gets 2x better. To use military terminology (ugh), it's a "force multiplier." It's not saying all our existing research is garbage. It's picking off the low-hanging fruit, the worst offenders, and thereby improving the signal to noise ratio of research overall. And once you get all the worst offenders, you can look for less low-hanging fruit.

I bet Gates would be happy to admit a lot of the research he funds is "low quality." That isn't a personal indictment against him. He'd probably argue that a lot is "medium quality" and a lot is "high quality." I wouldn't disagree. I'm sure there is some proportion in all three buckets.

[mike_hearn]

You're welcome, thank you for the discussion.

As someone who lives outside of America, I really wouldn't be so down on it as a country. I'm lucky to live in a very nice part of the world (Switzerland!) but even so, America is a country and culture I hugely admire. I've spent most of my career working for American firms because that's where the action is, that's where the bravest people tackle the hardest problems. Yes, America is a land of extremes and the lows can be low, but the compensation is that the highs are really high.

"For example, in every field, funding is corrupted. So I wouldn't say there are some fields that don't get much invalid research at all. Outright fraud is not the only kind of "invalid" research."

Ah, I see. Well, I'd say that the funding mechanism makes invalid research possible/easy but doesn't necessarily directly create it. The lights are out but someone still has to misbehave. In some fields there just isn't much incentive to do that, e.g. consider computer graphics or the papers that explore more efficient K/V stores or compilers. You could try and cheat in those papers but why bother? You'd just be undermining any possible future jobs in industry, so I find these papers to be pretty reasonable.

The corruption really seems to kick off in fields that can be twisted into forms of social control in some way, or where there's not much chance of ever getting a good job in the private sector. Social sciences are a great example but public health is the same problem. People see an opportunity to change the world through misrepresenting their science, they see that nobody will stop them, and so they take it. Power is the goal and the apathetic funders are the enabler. You can't change the world or control anyone via compiler research though, so it stays closer to the original ideals.

Now, I agree that if you wider the problem scope to include irrelevant research nobody cares about, then indeed every field has big problems with that. It's probably too much to ask people to care about both invalid and irrelevant research at once though.

"I bet Gates would be happy to admit a lot of the research he funds is "low quality." That isn't a personal indictment against him. He'd probably argue that a lot is "medium quality" and a lot is "high quality." I wouldn't disagree. I'm sure there is some proportion in all three buckets."

Well if you or anyone else can get him to admit that, it'd be a great start.

[javert]

> You could try and cheat in those papers but why bother? You'd just be undermining any possible future jobs in industry, so I find these papers to be pretty reasonable.

I was a grad student in CS for 8 years, but left without finishing the PhD. I wasn't in compilers, but that's a reasonable example because I was in some other "niche of a niche." Small community, fairly obscure.

Most of the grad students in my niche really just wanted to be professors. And the way to become a professor was to (a) publish a very high quantity of papers; and (b) make friends with all the senior people in our little niche.

The goal people had wasn't to amass power. It was just to get a tenure-track job. Somebody else in this comment section made a joke about Chinese students. The problem is not at all limited to them. But a tenure-track job at an American university really seems like heaven to someone who's made their way up from the bottom in China, for example. And also to some people from other parts of the world, including America. If publishing tons of low-quality papers is the path to that, and being buddy-buddy with other people, they go for it. It's a "I'll scratch your back, you scratch mine" environment, including regarding "peer" review.

There is a tradeoff between quality and quantity. I was doing empirical research. I couldn't compete with the people doing more pure mathematical/algorithm stuff. They could just spit out papers with some new obscure algorithm (that will never be used anywhere) and a proof of some of its properties. I would have been lucky to have 5 published papers at the end of my grad school career, but a good tenure-track candidate would have like 30.

The senior people, like my thesis adviser, more than enable this kind of behavior. They get grant money basically based on the quantity of papers published. I never understood why my adviser cared so much about grant money. Like, what drove him to put out a super high quantity of crappy papers, to have a ton of students, and get a lot of grant money? What's the point? I never understood it. I mean, he already had tenure. And there were lots of tenured profs I knew who actually just didn't care about grant money and publication count, and didn't do that. Which is great. But you know who all the grant money goes to, and then who has a huge "lab" with lots of students? The ones like my professor who really care about that sort of thing and optimize for it.

I think compilers produces much higher quality than my niche-of-a-niche (which I don't want to name, by the way). But I don't think an area like that is immune to the pressures I'm talking about. It's probably a bigger community (which helps), where the research just has a different dynamic. We could speculate why that might be. But nonetheless I would argue that every area of modern science suffers from the problem I'm describing, to a varying extent between countries and fields and sub-sub fields.

By the way, maybe people go into compilers as PhD students, wanting to go to industry... but the truth is, in computer science, getting a PhD is usually not advancing your career over just putting that many years directly into industry. If your planned route is PhD=>industry, it only makes sense if you want to be in an industry research position and if you care about that more than how much money you make.