[javert]

I appreciate your thoughts.

I agree that pushing for cultural/political change is a hard way to go given the populism in America. I'm an American. On a personal level, I think the country is so broken that it's not worth fixing, and would like to expatriate.

> Re: 60%, that's your personal estimate, right? My impression is that it varies a lot by field. In some fields you don't really get much invalid research at all, it's a curiosity. In other fields 100% of papers are useless because the underlying premises of the field are themselves wrong.

Yes, it was my estimate. But it's just a number I threw out there. I don't disagree with what you're saying here, but I'm more focused on the structural problems that exist to some (varying) degree in every field. For example, in every field, funding is corrupted. So I wouldn't say there are some fields that don't get much invalid research at all. Outright fraud is not the only kind of "invalid" research. It's still "invalid" if it's honest research but exploring the wrong path because there's some old fart at the funding agency who has a lot of friends who are exploring that particular path, which actually has been played out for 10 years already.

> Blog post about Bill Gates

I appreciate your sharing this. I think it's easy to construct this sort of narrative. That doesn't mean it's true. It might be. I don't know enough to tell.

I really think someone like Bill Gates could see the kind of research validation institute I'm proposing as a "win." I'm not saying all the research he's funding is worthless. I don't believe that, personally. I'm just saying, we can boost the quality and effectiveness of research if we have this kind of institute. Maybe our research process as a civilization gets 2x better. To use military terminology (ugh), it's a "force multiplier." It's not saying all our existing research is garbage. It's picking off the low-hanging fruit, the worst offenders, and thereby improving the signal to noise ratio of research overall. And once you get all the worst offenders, you can look for less low-hanging fruit.

I bet Gates would be happy to admit a lot of the research he funds is "low quality." That isn't a personal indictment against him. He'd probably argue that a lot is "medium quality" and a lot is "high quality." I wouldn't disagree. I'm sure there is some proportion in all three buckets.

[mike_hearn]

You're welcome, thank you for the discussion.

As someone who lives outside of America, I really wouldn't be so down on it as a country. I'm lucky to live in a very nice part of the world (Switzerland!) but even so, America is a country and culture I hugely admire. I've spent most of my career working for American firms because that's where the action is, that's where the bravest people tackle the hardest problems. Yes, America is a land of extremes and the lows can be low, but the compensation is that the highs are really high.

"For example, in every field, funding is corrupted. So I wouldn't say there are some fields that don't get much invalid research at all. Outright fraud is not the only kind of "invalid" research."

Ah, I see. Well, I'd say that the funding mechanism makes invalid research possible/easy but doesn't necessarily directly create it. The lights are out but someone still has to misbehave. In some fields there just isn't much incentive to do that, e.g. consider computer graphics or the papers that explore more efficient K/V stores or compilers. You could try and cheat in those papers but why bother? You'd just be undermining any possible future jobs in industry, so I find these papers to be pretty reasonable.

The corruption really seems to kick off in fields that can be twisted into forms of social control in some way, or where there's not much chance of ever getting a good job in the private sector. Social sciences are a great example but public health is the same problem. People see an opportunity to change the world through misrepresenting their science, they see that nobody will stop them, and so they take it. Power is the goal and the apathetic funders are the enabler. You can't change the world or control anyone via compiler research though, so it stays closer to the original ideals.

Now, I agree that if you wider the problem scope to include irrelevant research nobody cares about, then indeed every field has big problems with that. It's probably too much to ask people to care about both invalid and irrelevant research at once though.

"I bet Gates would be happy to admit a lot of the research he funds is "low quality." That isn't a personal indictment against him. He'd probably argue that a lot is "medium quality" and a lot is "high quality." I wouldn't disagree. I'm sure there is some proportion in all three buckets."

Well if you or anyone else can get him to admit that, it'd be a great start.

[javert]

> You could try and cheat in those papers but why bother? You'd just be undermining any possible future jobs in industry, so I find these papers to be pretty reasonable.

I was a grad student in CS for 8 years, but left without finishing the PhD. I wasn't in compilers, but that's a reasonable example because I was in some other "niche of a niche." Small community, fairly obscure.

Most of the grad students in my niche really just wanted to be professors. And the way to become a professor was to (a) publish a very high quantity of papers; and (b) make friends with all the senior people in our little niche.

The goal people had wasn't to amass power. It was just to get a tenure-track job. Somebody else in this comment section made a joke about Chinese students. The problem is not at all limited to them. But a tenure-track job at an American university really seems like heaven to someone who's made their way up from the bottom in China, for example. And also to some people from other parts of the world, including America. If publishing tons of low-quality papers is the path to that, and being buddy-buddy with other people, they go for it. It's a "I'll scratch your back, you scratch mine" environment, including regarding "peer" review.

There is a tradeoff between quality and quantity. I was doing empirical research. I couldn't compete with the people doing more pure mathematical/algorithm stuff. They could just spit out papers with some new obscure algorithm (that will never be used anywhere) and a proof of some of its properties. I would have been lucky to have 5 published papers at the end of my grad school career, but a good tenure-track candidate would have like 30.

The senior people, like my thesis adviser, more than enable this kind of behavior. They get grant money basically based on the quantity of papers published. I never understood why my adviser cared so much about grant money. Like, what drove him to put out a super high quantity of crappy papers, to have a ton of students, and get a lot of grant money? What's the point? I never understood it. I mean, he already had tenure. And there were lots of tenured profs I knew who actually just didn't care about grant money and publication count, and didn't do that. Which is great. But you know who all the grant money goes to, and then who has a huge "lab" with lots of students? The ones like my professor who really care about that sort of thing and optimize for it.

I think compilers produces much higher quality than my niche-of-a-niche (which I don't want to name, by the way). But I don't think an area like that is immune to the pressures I'm talking about. It's probably a bigger community (which helps), where the research just has a different dynamic. We could speculate why that might be. But nonetheless I would argue that every area of modern science suffers from the problem I'm describing, to a varying extent between countries and fields and sub-sub fields.

By the way, maybe people go into compilers as PhD students, wanting to go to industry... but the truth is, in computer science, getting a PhD is usually not advancing your career over just putting that many years directly into industry. If your planned route is PhD=>industry, it only makes sense if you want to be in an industry research position and if you care about that more than how much money you make.