[exporectomy]

I wonder if we should separate the roles of scientist and researcher. Universities would have generalist "scientists" who's job would be to consult for domain-specialized researchers to ensure they're doing the science and statistics correctly. That way, we don't need every researcher in every field to have a deep understanding of statistics, which they often don't.

Either that or stop rewarding such bad behavior. Science jobs are highly competitive, so why not exclude people with weak statistics? Maybe because weak statistics leads to more spurious exciting publications which makes the researcher and institution look better?

[civilized]

The scientific establishment will never be convinced to stop doing bad statistics, so "the solution to bad speech is more speech". Statisticians should be rewarded for effective review and criticism of flawed studies, and critical statistical reviews of any article should be easy to find when they exist.

This is sounding like a great startup idea for a new scientific journal, actually.

[vavooom]

I do enjoy the idea of a journal focused entirely on the review of statistical methods and underlying methodologies applied in modern day research. Could act as a helpful signal for relevant and applicable research.

[robertlagrant]

Just adding an Arxiv filter that allows me to set a minimum p-value or variation % would do it!

[neumann]

Every medical researcher I've worked with had a biostatistician on hand to handle the stats. As a aerospace engineer, I always had interesting discussions with them on the meaningfulness of a clinical study with 15 people, but have come to appreciate the massive difficulty in progressing medical research if everybody were to wait for a clinical trial with a 1000 patients.

[pas]

There's no problem with a n=15 study, the problem is that there isn't a proper effective process that aggregates these small studies and the designs and conducts n=1000 ones. What we have instead is academic peacocking. (Grant applications judged by other scientists who are also at the same time in the grant game.)

Of course this is somewhat a necessary consequence of having academic freedom.

[jltsiren]

Such staff scientist roles for people with particular methodological skills do exist. They are not particularly common, because there are a few issues:

1. Who will pay for them?

2. How do we make staff scientist roles attractive to people who could also get tenure-track faculty positions or do ML/data science in the industry?

3. How do we ensure that a staff scientist position is not a career dead end if the funding dries up after a decade or two?

The standard academic incentives (long-term stability provided by tenure, freedom to work on whatever you find interesting, recognition among other experts in the field) don't really apply to support roles.

[Robotbeat]

We exclude people who don’t publish. Papers tend not to publish stuff that isn’t a positive result.