[matthewdgreen]

Alternatively there is a baseline of fraudulent behavior in any human organization of 1-5% and since there are tens of thousands of high-profile researchers this sort of thing is inevitable. The question you should be asking is whether the field is able to correct and address its mistakes. Ironically cases like this one are the success stories: we don’t have enough data to know how many cases we’re missing.

[vintermann]

I don't think the baseline is the same. The more competition, the more temptation to cheat. When the margins to win are small enough, cheaters are disproportionately rewarded.

Think of Tour de France. Famously doping-riddled. There are a lot of clean cyclists, but they are much less likely to be able to compete in the tour.

You can fight cheating with policing: doping controls, etc. But as the competition gets more extreme, the more resources you need to spend on policing. There's a breaking point, where what you need to spend on policing exceeds what you get from competition.

This is why almost no municipalities have a free-for-all policy for taxis. There are too many people technically able to drive people for money. All that competition drives prices lower, sure, but asymptotically. You get less and less lower prices the more competition you pile on - but the incentives for taxi drivers to cheat (by evading taxes, doing money laundering as a side gig etc.) keep growing. London did an interesting thing - with their gruelling geography knowledge exam, they tried to use all that competitive energy to buy something other than marginally lower prices. Still incentive to cheat, of course, but catching cheaters on an exam is probably cheaper and easier than catching cheaters in the economy.

(Municipalities that auction taxi permits get to keep most of the bad incentives, without the advantage of competition on price.)

[hayd]

It's only a story because he's president, if he were only a researcher/professor this would not even be a story. This is NOT a success story, it shows that this fraudulent behavior is endemic and an effective strategy for climbing the academic ladder.

A success story would be this is exposed at large... we work out some kind of effective peer-reproduced tests... and the hundreds/thousands of cheating professors are fired.

[tptacek]

Endemic means "regularly occurring". How many examples of this kind of misconduct are you aware of? Ok, now, what's the denominator? How much research is actually conducted? I'm personally familiar with 3 fields (CS, bio, and geology) and what I've learned is that the number of labs --- let alone projects --- is mind-boggling. If your examples constitute 1% of all research conducted --- which would represent a cosmic-scale fuckload of research projects --- how much should I care about it?

[bhk]

BMJ: Time to assume fraud? https://blogs.bmj.com/bmj/2021/07/05/time-to-assume-that-hea...

Study claims 1 in 4 cancer research papers contains faked data https://arstechnica.com/science/2015/06/study-claims-1-in-4-...

[cycomanic]

So let's talk about misleading headlines and citations in journal articles. I would argue that arstechnica is one of the better news sources. Despite that, if we go to the article there is a link to that there has been "a real uptick in misconduct". Now if we click through that link, it does claim that there has been an increase in fraud as a lead in (this time without a link) but the article is about something completely different (i.e. that almost half the retracted papers are retracted due to fraud).

As an aside, the article cites that there have been a total of 2000 retracted papers in the NIH database. Considering that there are 9 Million papers in the database overall, that is a tiny percentage.

[bhk]

> ... if we click through ...

So you deflect from the entire content of the article with that distraction? And then an additional misdirection regarding retraction? Why?

[cycomanic]

> > ... if we click through ...

> So you deflect from the entire content of the article with that distraction? And then an additional misdirection regarding retraction? Why?

What do you mean? I take issue with the headlines and reporting. And I believe if one claims lack of evidence, sloppy evidence or fraudulent evidence one should be pretty diligent about ones one evidence.

Regarding the claims in the article. If you look at the 1 in 4 article you find that the reality is actually much more nuanced, which is exactly my point. The reporting does not necessarily reflect the reality.

If you call that deflection...

[zzzeek]

devil's advocate - '1 in 4 studies are fake, says "study"'

[moab]

So just because one person is cheating, it means all academics are cheating?

FWIW, most top-ranked CS conferences have an artifact evaluation track, and it doesn't look good if you submit an experimental paper and don't go through the artifact evaluation process. Things are certainly changing in CS, at least on the experimental side.

It's also possible that theorems are incorrect, but subsequent work that figures this out will comment on it and fix it.

The scientific record is self-correcting, and fraud / bullshit does get caught out.

[hayd]

It's not just "one person", there is wide-spread fraud across many disciplines of academia. The situation, of course, is vastly different across subjects/disciplines, e.g. math and CS are not really much affected and I would agree they're self-correcting.

I might agree they're self-correcting in the (very) long-term, but we're seeing fictitious results fund entire careers. We don't know the damage that having 20+ years of incorrect results being built upon will have... And that's not to speak of those who were overlooked, and left academia, because their opportunities were taken by these cheaters (who knows what cost that has for society).

[panarky]

The very fact that the fraud is discovered, that reporters amplify it, and that it can bring down the president of the university, is evidence to me that the system still works.

[wpietri]

Maybe? I'd want to see a clear model of flows and selection biases before I concluded that.

Another way to look at it: perhaps Tessier-Lavigne only got this scrutiny because he was president of the university. And the fact that they didn't guarantee anonymity when "not guaranteeing anonymity in an investigation of this importance is an 'extremely unusual move'" might be a sign that the scrutiny was politically diminished.

So it could be that most of the equally dubious researchers don't get caught because not enough attention is paid to patterns like this except when it's somebody especially prominent. Or it could be that this one was not as well covered up, perhaps because of the sheer number of issues. Or that the cross-institution issues made Stanford more willing to note the wrongdoing. Or that Stanford is less likely to sweep things under the rug because of its prominence. Or just that there was some ongoing tension between the trustees and the president and that this was an opportunity to win a political fight.

[nequo]

These are good points and hard to know. But the Retraction Watch is tracking stories of both mistakes and fraud in published research, across universities:

https://retractionwatch.com/

[mike_hearn]

A tenacious undergrad doing journalism as a hobby is not a system.

[panarky]

The fate of the world lies in the hands of the young and inexperienced.

Grad students, Supreme Court clerks, 19-year-old soldiers.

[quickcheque]

Sure, any system with a false negative and false positive rate 'works'.

[michael1999]

No. This level of scrutiny and diligence is rare, and was selectively applied based on the targets profile. The "field" did nothing about this over 20 years. A computer science freshman did this as a hobby, not as a participant in neuroscience.

Perhaps "nothing" is too harsh. Various people in the field raised concerns on several occasions. But the journals did nothing. The "field" still honoured him. And _Stanford_ did nothing (except enable him and pay him well) until public embarrassment revealed the ugliness.

[dmvdoug]

This is the important and troubling point. Everyone trumpets science as a model of a rational, self-correcting social enterprise. But we see time and time again that it takes non-scientists to blow the whistle and call foul and gin up enough outside attention before something gets done to make the correction. That puts the lie to the notion of self-correction.

[moab]

This is an issue at the department politics level. For the scientific field, once someone starts retracting papers (and arguably, even before this), everybody knows that you should take person X's papers with a huge grain of salt.

E.g., in math / theory, if someone has a history of making big blunders that invalidate their results, you will be very hesitant to accept results from a new paper they put on arXiv until your community has vetted the result.

So yes, I do trumpet science as a model of a rational, self-correcting social enterprise, at least in CS.

Other sciences like biology and psychology have some way to go.

[VHRanger]

The thing is that replication is inherently easy in CS. Especially now that people are expected to post code online.

Forcing authors to share raw data and code in all papers would already be a start. I don't know why top impact factor papers don't do this already.

[moab]

I completely agree. It's a pity that this isn't becoming standard in fields affected by the replication crisis. I would be happy to be corrected if someone has heard / experienced otherwise.

[chasd00]

> you will be very hesitant to accept results from a new paper they put on arXiv until your community has vetted the result.

Forgive my ignorance but I thought that was SOP for all papers. Is it not?

[moab]

Well not really, right? Let's suppose some well known, well respected author that has a history of correct results puts up a new paper. I (and I think most people) will assume that the result is correct. We start to apply more doubt once the claimed result is a solution to a longstanding open problem, or importantly, if the researcher has a spotty track record for correctness (in math/TCS) or falsifying results (in experimental fields).

But really we shouldn't be talking about math errors and falsification in the same category.

[karaterobot]

The problem is that we don't know what the baseline really is. We know that between a third and a half of results from peer reviewed papers in many domains cannot be replicated. Looking closer, we see what look like irregularities in some of them, but it's harder to say which of them are fraud, which are honest mistakes, and which of them just can't be replicated due to some other factors. But because so many of these studies just don't pay off for one reason or another, I would agree that it is getting really hard to rely on a process which is, if nothing else, supposed to result in reliable and trustworthy information.

[cycomanic]

Where is that number of 1/3-1/2 coming from? And which fields? I find that very hard to believe (at least if we exclude the obvious fraudulent journals, where no actual research gets published)

[zeroCalories]

I think he's referencing the replication crisis that was a big deal a few years ago. Psychology was hit hard(unsurprising), but a few other fields in the biology area were also hit.

[karaterobot]

It's worst in Psychology and the Social Sciences, but it's not limited to them. Per Wikipedia:

> A 2016 survey by Nature on 1,576 researchers who took a brief online questionnaire on reproducibility found that more than 70% of researchers have tried and failed to reproduce another scientist's experiment results (including 87% of chemists, 77% of biologists, 69% of physicists and engineers, 67% of medical researchers, 64% of earth and environmental scientists, and 62% of all others), and more than half have failed to reproduce their own experiments. But fewer than 20% had been contacted by another researcher unable to reproduce their work. The survey found that fewer than 31% of researchers believe that failure to reproduce results means that the original result is probably wrong, although 52% agree that a significant replication crisis exists. Most researchers said they still trust the published literature

Not sure if the results of that online study have (or can) themselves be reproduced, however. It's turtles all the way down.

[zeroCalories]

Skimmed the wiki on the replication crisis, and people have actually tried to systemically replicate popular studies and found similar results. You could say there has been a successful replication of failure to replicate.

[whydoyoucare]

If a field takes two decades to "correct" its mistakes, then there are several things wrong with it. And if we have top positions held by unethical people, who have got away with it, and possibly climbed to the top because of it, then I do not know what to feel or say about this.

[FrustratedMonky]

It's taken String Theory a few decades to correct itself.

[jlawson]

Any human organization?

I don't expect 1-5% fraud in airline pilots, bank tellers, grocery store clerks, judges, structural engineers, restaurant chefs, or even cops (they can be assholes but you don't have to bribe them in functional countries).

I think academics can do better than 1-5% fraudulent.

[cycomanic]

What? In all of the ones you mentioned there is a known significant amount of fraudulent behaviour.

Store clerks, theft is about 1-2% of sales typically. It has been said for years that the majority of that theft is from employees. Airline pilots have been known to drink during their flights (or go away from there seat for other reasons that are not in the rules).

Cops, I mean don't get me started, just the protection of a cop who has done something wrong by the other cops would count as fraudulent, but I don't see many cops going after their own black sheep.

Judges, in Germany deals (i.e. the accused pleads guilty to lesser charges so the bigger ones get dropped) are only legal under very limited circumstances (almost never and need to be properly documented). Nevertheless, in studies >80% of lawyers reported that they had encountered these deals).

I think you seriously underestimate the amount of fraudulent behaviour.

[cycomanic]

Also coming back to judges. The behaviour by Thomas and Alito regarding presents etc. Would count as serious scientific misconduct in academia. So there's a significant percentage just there already.

[P_I_Staker]

I expect far higher levels of fraud in these professions.