[Bakkot]

Actually, that would have been way better. If they had done the study you suggest, and the result had still been significant, then he would have been entirely justified in reporting what he did.

The issue is that dividing the participants after the fact and then looking for correlation in the existing data reduces the significance of the statistic considerably (we have other statistics for that). The p-value is not representative when used that way.

But if you do another study focused on that group in particular and still get a significant result, you're fine! The problem isn't that they located a group on which the drug worked in a dishonest way, or some such - the problem is that they were dishonest to claim they had significant evidence that the drug worked on that group. If they'd done an additional study on that group in particular, they would have their evidence (or, of course, a null result).

[andrewcooke]

i think the person you were replying to was implying that the same data be used, while what you are arguing is that there should be new observations made (and i agree with you, if the new work is independent; i just wanted to explain why i think the original comment was arguing for greater transparency).

[Malician]

I believe he's suggesting that the doctor could have legally covered up the results of the first trial (by simply not releasing them,) then run a second trial on only the most beneficial population, releasing those results without mentioning the first trial.

At that point, his product would look great, hiding its failures.

This way, while he misinterpreted the P value in an illegal and fraudulent way, he did release all relevant information - ironically, better for the informed reader than if he had rerun the trial legally.

[feral]

Malician, I'm not sure you and DanBC understand this fully?

It would be absolutely fine to run a new trial on the supposedly most beneficial population (those with mild/moderate lung damage; lets call them 'the subpopulation').

If that second trial succeeded, then it would be strong evidence that the drug was beneficial for the subpopulation.

There would be no need to hide the results of the first trial, as the first trial did not provide evidence that the drug didn't work on the subpopulation.

If you read the article to the end, they did in fact do such a trial on the subpopulation. And they got evidence it wasn't working on the subpopulation - which is how science goes.

The problem was that the first trial wasn't set up to examine the subpopulation, but they reported results as if it was. You can't do that with standard NHST, as it invalidates the assumptions of the statistical framework being used.

But you can absolutely decide to run a whole new test on a new sub population, based on hints you get from the first results.

And, while it'd in general be better if all test results (positive AND negative) were published, that is not relevant to this situation - the first trial said nothing bad about the effects on the subpopulation, so there'd be nothing to gain from hiding it, if you just wanted to claim it worked on the subpopulation.

Its not like a situation where they got evidence that the subpopulation would not benefit in the drug in the first test, and then decided to do another test, planning to only report the second.

[Malician]

Yes, I understand this. This is correct if the result of the test on that said subpopulation is only interpreted by the public and/or scientific community as applying to the subpopulation.

However, if the results of the original test are hidden, the results of the second test could well be taken as evidence for a wider or stronger effect, yes? If this isn't the case, then I wouldn't see a problem with that behavior - but from the reading I've done, I suspect it is in fact the case and is common practice.

edit: I may be completely wrong on this - if, indeed, that's not a significant problem.

[andrewcooke]

ah, ok. so, you're right, but not as right as th eoriginal issue being discussed :o) i can explain if you're interested...

what i think you're saying is that they would hide the original negative study and publish a subsequent (new, separate, on different people) positive study.

[aside - that's not a perfect description because for one particular group the first study was positive; it's just that the group in question wasn't explicitly targetted].

and, in general, that's considered a bad thing. because (1) you can keep repeating studies until you get a positive, and then publish and (2) because the negatives aren't published, people have incomplete information.

but it's not a terribly bad thing, because if something isn't true then, if you repeat a study, it's likely going to show it isn't true. the standards are set high enough that you'd need to do hundreds of studies before you showed something to be true (when it really isn't).

and because hundreds of studies are expensive, it's unlikely to happen (but then you think of the industry as a whole, and it is doing hundreds, and so some of those are likely wrong...).

in contrast, what this guy was prosecuted for was hunting in the data. you can think of that like doing a new study, but without the cost. it's pretty easy to dream up hundreds of different questions you can ask existing data. and just by luck you're going to find the occasional surprising answer.

so hunting through data is like doing hundreds of studies until you find something, but it's cheap! and that's why it's "worse" than simply hiding negative results and repeating studies. because it's much more likely to happen in practice.

[pessimizer]

I think you're getting hung up on their use of the word "hide." What they're saying is that the first study could have been disregarded except as a good reason to run the second study. Of course, that later happened, and the effect disappeared - but maybe it wouldn't have. That's how science works.

I don't thing that you're disagreeing with them, just reiterating.

[PeterisP]

It would actually be ok to do what you say - the problem is that there WASN'T a 'most beneficial population' for which his treament works; he made up that 'population' from data which, at best, show that this population could be more beneficial if it's verified.