[stevenschmatz]

I did research at a lab at Harvard Medical School. I can verify that the "publish-or-perish" mindset of academia drives researchers to do anything to get that significant p-value – even at some of the most highly regarded and established institutions.

Science today is full of bureaucratic nightmares - the publishing of a large amount of trivial results and the transformation of the most experienced scientists to uninvolved managers.

[BenoitEssiambre]

I have the same experience.

Until university positions and research grants stop being given out based on prior research results we won't be able to trust the research performed there.

There are millions of dollars on the line for researchers involved. It is the difference between a well paid career or a life of destitution while being a slave to huge student debt. It's no wonder they are blind to the flaws in their research.

I believe strongly that teaching positions and research grants should be given out based on criterions that are only incidental to research results.

Evaluate profs and grants based on:

1. domain knowledge (test the applicants) 2. math skills (test the applicants) 3. motivation and leadership 4. prior and current research proposals (but ignore results, especially the fact that they were published or not). 5. other skills such as written and oral communication

Universities should not rely on journals to evaluate their professors. This corrupts the whole system. Journals have different goals. They want to publish well done research with interesting results. Universities should hire researchers that do well done research with interesting _questions_ regardless of the results.

If universities keep giving out jobs based on having generated interesting results in the past, they are going to keep getting researchers that ignore biases and publish whatever results are interesting whether they are true or not.

[tjradcliffe]

I don't know that the criteria you're proposing are all that different from the current situation. Domain knowledge and math skills are tested by your eduction. Motivation and leadership by committee work. Other skills by teaching. The only difference is your focus on proposals rather than results, and this is already the case in some fields.

I come for a "search for physics beyond the standard model" background, where other than the neutrino mass (from the SNO collaboration, which I was part of) there hasn't been a positive result in decades. So there is already a good deal of focus on proposals rather than results, and yet almost all the issues I see in the biosciences (I jumped ship to genomics in the mid-00's) are also present in that area of physics.

Ergo, empirically, I'm doubtful that focusing on proposals rather than results will make much difference.

The difficulty is that science never makes economic sense for an individual. I spent a decade of my life measuring zero to higher and higher precision, and I know people who have spent entire careers doing so: putting new limits on branching ratios to exotic (which sounds so much better than "nonexistent") decays and so on. It was fun, although I took a year off in the middle to do some medical physics and imaging, which was even more fun because I actually got to measure phenomena that exist.

So when I read things about the paucity of "breakthrough discoveries" I think that mostly the low-hanging fruit have been picked and genomics turns out to be a whole lot harder and more of a slog than people expected, with a vast amount of uninteresting material to be waded through for the sake of a slow accumulation of knowledge that we are still a century away from putting to any very good use.

I don't know what an economically rational model for reward in such an environment is, and it's good that the article raises the issue and explores some alternative approaches, but I don't think there is any easy fix for the problem because I don't think science makes any economic sense. Just moral sense.

[jules]

That way you'll get loads of crap research too. In fact if you evaluate scientists on criteria such as "motivation and leadership" you make it more political not less. In your system there is absolutely no incentive to actually do the research, so you'll give all the money to a bunch of people who are great at writing proposals but who don't actually do science. Every second spent doing science is a second not spent writing the research proposals.

There is a very simple solution to the problem that completely eliminates gaming research results and publishing bias. Require that the statistical methodology is completely specified prior to any data acquisition. The paper is written before the data is acquired and it has some blank spots where the data will be filled in with a method that is completely mechanical (e.g. with a computer program that processes the data and spits out the figures that will be used to fill the blanks). Journals should decide whether to publish a paper or not based on the version without the data.

[pcrh]

>but ignore results

That is certainly an interesting proposal. How do you intend to assess competence in generating novel ideas (i.e. not testing for knowledge of existing work) if you ignore the candidate's track record?

[chrisseaton]

If you evaluate faculty and grants on research proposals but not the results, what stops a researcher making lots of big idea proposals, but never actually doing any work? At some point, someone needs to actually do the experiments. In your system there is no motivation to do that.

What I think we need to do is reward negative results as much as positive results.

[ylem]

Student debt? But for grant review (I have reviewed NSF proposals before), I'd say that one of the biggest challenges is that there are a lot of people chasing a rather small amount of money (in the physical sciences). I think our committee had several strong people on it for the field and even if people were not listed as having a conflict of interest, they would volunteer if they had one and tried to be fair.

However, when you have a small pot of money, you do have to think about how do you make awards. On the one hand, you do ask the question, what are the chances that this would work and if it did, would it be "transformational"? That is, it may turn out to be a loss, but if it works, then it could really advance the field--in our committee, we did try to fund those kind of proposals over incremental advances.

Now, as to the question of reputation, I'm going to have to disagree. If you're going to give someone funds, how do you gamble? If they're young, then you can just look at their idea, their resources, and some indication that they have a chance at success. However, for a more senior researcher, they do have a track record. If they've received funds in the past and haven't accomplished anything with them, then why would you keep giving them money? If someone is publishing interesting results, then other people will try to duplicate and extend them If someone's research consistently fails these tests, then their reputation will suffer.

As for domain knowledge/math skills--I have to say that I think that this is relatively useless. Do you have any feeling for how many grant applications come in (along with multiple proposers)? And you want to test them across many subfields, etc.? These people managed to get their phDs, so if they are not competent, then that should have either showed up earlier, or in their publications.

I think there's a lot of merit in judging people by their results rather than simple tests which could be gamed.

[untilHellbanned]

> If universities keep giving out jobs based on having generated interesting results in the past

Worse they are being given out almost exclusively based on where those past results are reported...

[memming]

And yet, I have high hopes in science. It is not the most efficient system, but it is the only system that is consistently working well to advance our knowledge.

We definitely have a lot of room to improve though.

[tedks]

As someone inside academic science, I think the only good thing it does is give people a place to think for a few years with relatively few distractions before they can go into industry or form a startup and actually get things done.

If you want to consider a problem in great depth before launching a startup to attack it, definitely go to grad school and do science for 4-5 years. If you think you have a good idea forget about grad school and just launch. (Obviously doesn't apply to things like bio where you need a lot of equipment, but as DIY science becomes more tractable this will go away.)