[BenoitEssiambre]

That is incorrect. The title is completely wrong.

"Study failed to find a reduction" _does not equal_ "There wasn't a reduction".

From the abstract: "The risk of death from colorectal cancer was 0.28% in the invited group and 0.31% in the usual-care group (risk ratio, 0.90; 95% CI, 0.64 to 1.16)"

Look at the confidence interval. This study was both consistent with a 36% reduction in deaths or a 16% increase in deaths. It's just a really wide range. All we can say about this study is that it didn't gather enough data to identify the size of the effect, not that there wasn't an effect.

This is likely because there wasn't that many people who died of colon cancer in any of the groups. This study just didn't track enough people to provide an answer.

[timr]

> That is incorrect. The title is completely wrong....Look at the confidence interval. This study was both consistent with a 36% reduction in deaths or a 16% increase in deaths. It's just a really wide range. All we can say about this study is that it didn't gather enough data to identify the size of the effect, not that there wasn't an effect.

In a randomized controlled trial you either find a significant difference in your metrics, or you don't. There's no other option. In all cases where you don't find a significant difference, the problem is that the confidence interval is too wide for whatever difference seems to exist. Your argument here is a fallacy (i.e. "you just didn't do a big enough sample!") which is a variant of my personal favorite: "it would have worked if you'd done X, Y or Z!"

There's always another X, Y, or Z. The negative study is always too small for the people who believe in the thing it's testing. As a supporter of some intervention, the onus is therefore on you to prove your claim in a demonstrated scenario, not on everyone else to disprove it in all scenarios. Could it be true that colonoscopies have some significant benefit to mortality smaller than detectable by a 80,000-person RCT? Sure. But that doesn't make the headline wrong.

This study didn't find a mortality benefit. Arguing that there's some theoretical other study that might find a benefit isn't relevant.

[BenoitEssiambre]

> In a randomized controlled trial you either find a significant difference in your metrics, or you don't. There's no other option.

This is a poor way of thinking about statistics. Whether you reject or not a sharp null hypothesis doesn't give you much information (See for example: https://www.gwern.net/Everything). Failing to reject in particular, can be compatible with a wide range of effects.

>In all cases where you don't find a significant difference, the problem is that the confidence interval is too wide for whatever difference seems to exist.

With enough data, there could totally have been a tight range around no effect or a small effect. This is not what we got here though.

Also note that other variables such as cancer risk came out significant, so while this study doesn't provide much inductive evidence around cancer death, we do get some deductive evidence based on the known link between cancer and death. Not to mention that cancer and cancer treatments are not fun even when they don't kill you.

[timr]

> With enough data, there could totally have been a tight range around no effect or a small effect. This is not what we got here though.

What the trial showed was a small effect with a wide uncertainty on a big sample. We cannot distinguish this from zero.

Again, could the observed effect be significant with a larger trial? Sure. But that's always true for a negative result. The objection carries no information.

[BenoitEssiambre]

>could the observed effect be significant with a larger trial? Sure. But that's always true for a negative result.

Sure, this is true, it's one of the reasons why results being significant or not is not very relevant. At some point you want to move towards whether the effect size is in a clinically relevant range or not.

>The objection carries no information.

Inasmuch as something like a confidence interval provides an idea of the range of the effect size, more data does carry more information. I know it's complicated to do this analysis properly with prediction intervals and such, but you have no choice if you want to be able to make good decisions with your data. A wide range estimate that doesn't allow you to make good clinical decisions is not useful.

For clinical purposes, I would even have been more confortable treating an significant but small effect in support of the "let's not test" scenario, than this wide range where the effect could be large and positive or negative on the other side and we just don't know. Significant doesn't automatically mean "do the test" and vice versa. Effect size matters! A non-significant result because of a wide interval just doesn't tell you much useful information.

[timr]

> Sure, this is true, it's one of the reasons why results being significant or not is not very relevant.

No. Significance is the only thing that matters here. If you don't have a significant result, you don't have a result. Making up stories about how the results coulda-woulda-shoulda been significant if only the study was different somehow is fine for bedtime or planning the next study, but absolutely irrelevant to interpreting the clinical trial in front of you.

The CI here is not actually that wide; I was being colloquial. It's an 80,000 person trial, with 40,000 per arm. The absolute observed difference in colo-rectal mortality between the two arms was 0.03%. The per-protocol analysis (just those people who got tested) was a difference of 0.15%.

That latter figure is the best possible argument for colonoscopy, and no matter how you look at it, it's just not a big difference. Even if you ran a huge trial to get a significant result at these effect sizes, you're still talking about a difference of 15 people per 10,000 (at best) screened. That's a lot of pain and expense for very little gain.

[phkahler]

I'm also confused about the "invited" group. Not all of them had the procedure and there is a part talking about lower cancer rates among the subset that actually accepted the invite. It sounds like there is still confusion about how to interpret it.

[timr]

There's not confusion. The study did an intention-to-treat analysis, which failed to find a significant result.

"Intention to treat" here means that you count everyone in the group that got an invitation to get a colonoscopy, regardless of whether or not they actually did it. Though this sounds counterintuitive, it's the "gold standard" because, if you don't do this, you leave yourself open to bias -- maybe the people who seek out colonoscopy have some symptom, family history or other reason that leads them to seek out treatment. Maybe the people who get a test get more treatment, and that treatment is harmful in the marginal case. Or just as importantly: maybe the people who don't have the time/inclination to do one would be better served by an alternative test.

Everyone (including GP) is fixating on the magnitude of the primary outcome and squabbling about whether or not colonoscopies help people. But I think the more interesting aspect of this study is that it shows that the genetic tests probably aren't inferior to the invasive, painful, time-consuming rectal exam. If that's true, it's great news!

[dahart]

> Though this sounds counterintuitive, it’s the “gold standard” because, if you don’t do this, you leave yourself open to bias

Isn’t there just as much chance for bias if the treatment is voluntary? Maybe the people who are more likely to have health issues are less likely to treat them.

I think there is a valid question about how effective a colonoscopy is given that you get one, and a separate valid question about how effective telling people to get colonoscopies is. According to the article, this paper answer the second question strongly via “gold standard”, and the first question less strongly via secondary analysis.

Part of the reason it’s counterintuitive here is the title of the article is “effect of colonoscopy screening”, not “effect of a doctor’s invitation to have a colonoscopy screening”. The title more than suggests that we’re comparing the outcomes of actually having the screening to not having one.

> maybe the people who don’t have the time/inclination to do one would be better served by an alternative test.

I see what you’re saying - the overall effectiveness of our current system may be low because of a low rate of voluntary adoption of the colonoscopy is low, and even a lower accuracy screen could be more effective if more people opt in.

One problem with drawing a conclusion this way is it ignores the possibility for dramatic changes in either opinion or in procedure of colonoscopies. What if we had the tech to do the colonoscopy at home in private? Would that change the voluntary rate of testing dramatically?

[timr]

> Isn’t there just as much chance for bias if the treatment is voluntary?

I'm not quite sure what you're asking here. If you're wondering if voluntary opt-out of colonoscopy carries risk of bias, then I'll say the following: it's an intervention that is painful, intrusive and time consuming. No reasonable person would get one absent demonstrated benefit.

Pick a thing where people are reasonably likely to do it as default behavior (eating chocolate, say), and the intervention is to abstain from doing the thing, then you'd be right to ask that question. I imagine people who voluntarily abstain from chocolate are pretty different in substantial ways than people who have to be coerced to do so. But people who don't get a colonoscopy when not pestered to do so are just...normal.

> I see what you’re saying - the overall effectiveness of our current system may be low because of a low rate of voluntary adoption of the colonoscopy is low, and even a lower accuracy screen could be more effective if more people opt in.

Not quite. I'm saying that this study did the fairest possible test for effectiveness of colonoscopies, and the effect sizes they found were on par with the genetic tests (to be clear: they didn't actually make this comparison in the RCT; I'm extrapolating from other studies.)

The evidence presented here is not that the genetic tests are a "lower accuracy screen", it's that colonoscopies are likely not better than genetic tests. That's very different.

[dahart]

> No reasonable person would get one absent demonstrated benefit.

But if we insist on measuring demonstrated benefit by factoring in participation rates, then it’s a catch-22. What if given a prior distribution of 100% opt-in, colonoscopies are effective? And you ignored my question about what if we made colonoscopies more convenient, less intrusive and time consuming, which is becoming possible with new tech. These things can change the participation rate, which in turn can flip the outcome from little demonstrated benefit to high and conclusive demonstrated benefit.

> this study did the fairest possible test for effectiveness of colonoscopies

“Fair” is a subjective term, and it depends on what question you’re asking. I agree with your statement if the question is how effective is the current system of recommending colonoscopies. It’s not the fairest test of how effective a colonoscopy screen could be if everyone shows up for the screen. Colonoscopies might be not better than genetic tests because participation rates for genetic tests are higher, as opposed to colonoscopy screening being less effective on their own.

I understand your point that the total probability is important. But so is understanding the Bayesian factors, it’s equally enlightening and important to separate and understand the effectiveness of the screen given participation, from the likelihood of participation. And you effectively cemented how important this point is by clarifying that people use knowledge of these outcomes in order to choose whether or not to participate, so framing them incorrectly can and likely does lead to unnecessary loss of life.

[timr]

> What if given a prior distribution of 100% opt-in, colonoscopies are effective?

That is what the "per-protocol" analysis in this RCT estimates. They considered only those people who had a colonoscopy. This completely breaks randomization and is subject to investigator bias, but at least it gives you an idea of the best you could possibly do if you lived in a world where everyone was forced to get one.

And that is: a drop of 0.15% in colo-rectal death. They then attempt to extrapolate this to estimate the effect on all-cause mortality if you somehow forced everyone to get a colonoscopy, and come up with a number of 10.88% (vs. a baseline of 11.03%). So, even if you forced everyone to get a test, you'd need to test 667 people to save one life.

That's a lot of pain for very little gain.

[jdeibele]

My father died of colon cancer at age 67. I've been getting screened every 5 years, first by sigmoidoscopy and the last couple of times with the full colonoscopy.

With a sigmoidoscopy you're awake and the doctor will show you what they're looking at. I guess that's intrusive but it certainly wasn't painful.

With colonoscopy, you're under anesthesia. It was probably intrusive but since I wasn't conscious, I didn't care. There was no pain when I regained consciousness.

I'd rather do a stool sample by mail or dropping it off at the clinic if it has the same results as the colonoscopy. There's always a risk with general anesthesia.

I don't know if my case is the norm and yours is the exception. I tend to think it is. My dad missed spending time with his grandchildren and it's possible he'd still be around if he'd been examined. So get that colonoscopy.

[timr]

I grant you that I'm using an expansive definition of "pain" here...I'm including the day of diarrhea before the test, the stress and (yes, some) pain of the sedation itself, and the unpleasant recovery (nausea, fatigue, etc.) that follows. Maybe it isn't "pain", per se, but it's something we'd all rather avoid.

I should also say the following: if you've got a family history of colon cancer, active symptoms, or some other reason to believe that you're at risk, studies of statistical averages don't apply to you.

[JoeAltmaier]

I never have anaesthesia for colonoscopy. The post-procedure nausea etc are worse than the discomfort of being probed. I you're not squeamish (and I am not) then it's not a big deal to be awake for the procedure.

[SoftTalker]

There's not only a risk with the anesthesia but there's a risk of a perforated colon. It's not the routine risk-free procedure that the providers make it out to be.

[2devnull]

The important part of your post, “If that’s true…”

Is it true, or not?

[timr]

To the extent that we can assess it by this single study, yes.

There is no answer to the question you're asking. You're seeking absolute certainty where none can be had. We only know what we know as far as we know it. Always and everywhere.