Hacker News new | ask | show | jobs
by InSteady 951 days ago
Ah, the game of telephone.

>For more than 150 trials, Carlisle got access to anonymized individual participant data (IPD). By studying the IPD spreadsheets, he judged that 44% of these trials contained at least some flawed data: impossible statistics, incorrect calculations or duplicated numbers or figures, for instance. And in 26% of the papers had problems that were so widespread that the trial was impossible to trust, he judged — either because the authors were incompetent, or because they had faked the data.

Firstly, this is only from one journal, Anesthesiology. Second, the phrase "at least" indicates that while 44% had some amount of (presumably) flawed data, only 26% of the studies were bad enough to be judged fake or severely flawed by this one (admittedly esteemed) researcher in the field of anesthesiology. It's important to be skeptical and do your homework when you hear sweeping and/or shocking results. It's also important to read carefully, especially with science journalism because it is written for clicks and broad audiences, not to reduce ambiguity and adhere to strict standards of accuracy.

1 comments

I didn’t go look up the quote but based on your version here it sounds like roughly half (44%) had some kind of suboptimality, and of those roughly a quarter (26%) had serious problems preventing them from being relied on.

That means 11% of the total papers should be discarded, which means 89% of the papers can be used.

As per usual, science reporting fails to use precise language. It can be interpreted either way, although I think your interpretation is the slightly larger leap based on phrasing. In any case, it is far below the 70% (and not directed broadly at all scientific research) that GP states.
I went and read the article but have not tracked down the original paper.

It is at least 26%, because that was the percentage of studies that provided access to their data that proved faked or fatally flawed.

It may be substantially higher.