[DavidSJ]

The article seems to suggest the false positive rate is only 38%:

The trial followed 25,000 adults from the US and Canada over a year, with nearly one in 100 getting a positive result. For 62% of these cases, cancer was later confirmed.

(It also had a false negative rate of 1%:)

The test correctly ruled out cancer in over 99% of those who tested negative.

[hn_throwaway_99]

If the stats were as good as the hyperbole in the article, it would clearly state the only 2 metrics that really matter: predictive value positive (what's the actual probability that you really have cancer if you test positive) and predictive value negative (what's the actual probability that you're cancer free if you test negative). As tptacek points out, these metrics don't just depend on the sensitivity and specificity of the test, but they are highly dependent on the underlying prevalence of the disease, and why broad-based testing for relatively rare diseases often results in horrible PVP and PVN metrics.

Based on your quoted sections, we can infer:

1. About 250 people got a positive result ("nearly one in 100")

2. Of those 250 people, 155 (62%) actually had cancer, 95 did not.

3. About 24,750 people got a negative test result.

4. Assuming a false negative rate of 1% (the quote says "over 99%") it means of those 24,750 people, about 248 actually did have cancer, while about 24,502 did not.

When you write it out like that (and I know I'm making some rounding assumptions on the numbers), it means the test missed the majority of people who had cancer while subjecting over 1/3 of those who tested positive to fear and further expense.

[inglor_cz]

"only 2 metrics that really matter"

Nope, there is another important thing that matters: some of the cancers tested are really hard to detect early by other means, and very lethal when discovered late.

I would not be surprised if out of the 155 people who got detected early, about 50 lives were saved that would otherwise be lost.

That is quite a difference in the real world. Even if the statistics stays the same, the health consequences are very different when you test for something banal vs. for pancreatic cancer.

[tptacek]

Careful: the stats you're reading are all-cancers, cancers aren't uniformly prevalent, and the specific cancers you're referring to might (and probably do) have much, much worse screening outcomes than the aggregate.

[inglor_cz]

"and probably do"

Why probably?

I don't see where this "probably" comes from; it could well be the other way round. It is a new technology and its weak and strong points / applications may differ significantly from what we currently use.

[tptacek]

You're overfocused on the "technology" and underfocused on the base rate of the cancers you're concerned about. Just do the math. What you need to know is "just how accurate would this test need to be in order for most of the positives it generates to be true positives". The numbers will be surprising.

[inglor_cz]

Both the technology and the base rate DO matter.

Say that you are hunting the elusive snipe, one bird in a million. With standard techniques, you will have a lot of false positives.

But if you learn that the elusive snipe gives off a weird radio signal that other birds don't, your hunt will be a lot shorter.

Same with relatively rare cancers. If you can detect some very specific molecule or structure, your test will be quite reliable anyway. That is why I don't get your use of "probably". Unless you are really familiar with the underlying biochemistry, the probabilities cannot be guessed.

There is absolutely no reason why tests for rare diseases should have high false positive rates. In many other diseases, they don't. For example (although the underlying technology differs), Down syndrome is rare, but its detection barely has any false positives. You can test the entire pregnant population for Downs reliably, and many countries already do that.

[holowoodman]

Maybe. Let's do a thought experiment.

Let's say you do have a positive test for pancreatic cancer. Overall 5 year survival rate 12%, but other than with other cancers, people continue to die after that. Basically, it is almost a death sentence if it is a true positive. Early detection will increase your odds a bit, and prolong your remaining expected lifetime, but even stage 1 pancreatic cancer, only 17% survive to 10 years. Let's say you are one of the 99% of false positives, because everyone gets tested in this hypothetical scenario. Let's say imaging and biopsy looks clean. No symptoms (which you typically don't have until stage 3 with pancreatic cancer, where it is far too late anyways). With the aforementioned odds, what would you do?

Panic? Certainly, given that if it is a real positive, you might as well order your headstone.

Panic more? Maybe people with those news will change their behaviour and engage in risky activities, get depressed, or attempt suicide (https://jamanetwork.com/journals/jamanetworkopen/fullarticle... ). All of which will kill some of those people.

Get surgery to remove your pancreas? Well, just the anesthesia as a 0.1% chance of killing you, the surgery might kill 0.3% in total. No pancreas means you will instantly have diabetes, which cuts your life expectancy by 20 years.

Start chemotherapy? Chemo is very dangerous, and there is no chemo mixture known to be effective against pancreatic cancer, usually you just go with the aggressive stuff. It is hard to come by numbers as to how many healthy people a round of chemo would kill, but in cancer patients, it seems that at least 2% and up to a quarter die in the 4 weeks following chemotherapy (https://www.nature.com/articles/s41408-023-00956-x ). And chemotherapy itself has a risk of causing cancers later on.

Start radiation therapy? Well, you don't have a solid tumor to irradiate, so that is not an option anyways. But if done, it would increase your cancer risk as well as damage the irradiated organ (in that case probably your pancreas).

So in all, from 100 positive tests you have 99 false positives in this scenario. If just one of those 99 false positives dies of any of the aforementioned causes, the test has already killed more people than the cancer ever would have. Even if no doctor would do surgery, chemotherapy or radiation treatment on those hypothetical false positives, the psychological effects are still there and maybe already too deadly.

So it is a very complex calculation to decide whether a test is harmful or good. Especially in extreme types of cancer.

[inglor_cz]

"Let's say you are one of the 99% of false positives, because everyone gets tested in this hypothetical scenario."

This alone is a disqualifier for your scenario. A test with 99 per cent of false positives will not be widely used, if at all. (And the original Galleri test that the article was about is nowhere near to that value, and it is not intended to be used in low-risk populations anyway.)

I am all for wargaming situations, but come up with some realistic parameters, not "Luxembourg decided to invade and conquer the USA" scenarios.

[holowoodman]

> Nope, there is another important thing that matters: some of the cancers tested are really hard to detect early by other means, and very lethal when discovered late.

You are arguing for testing everyone there. If you cannot detect them by other means, you need to test for them this way. And do it for everyone. You have already set up the unrealistic wargaming scenario. You picked pancreatic cancer as your example where you do have to test every 6 months at least, because if you do it more rarely, the disease progression is so fast that testing is useless. There are no specific risk groups for pancreatic cancer beyond a slight risk increase by "the usual all-cancer risk factors". Nothing to pick a test group by.

And a 99% overall false positive rate is easy to achieve, lot's of tests that are in use have this property if you just test everyone very frequently. Each instance of testing has an inherent risk of being a false positive, and if you repeat that for each person, their personal false-positive risk of course goes up with it. All tests that are used frequently have an asymptotic 100% false positive rate.

[inglor_cz]

"You are arguing for testing everyone there."

Are you mistaking me for someone else? I never said or even implied that.

"And a 99% overall false positive rate is easy to achieve,"

Not in the real world, any such experiment will be shut down long before the asymptotic behavior kicks in. Real healthcare does not have unlimited resources to play such games. That is why I don't want to wargame them, it is "Luxembourg attacks the US scenario".

"There are no specific risk groups for pancreatic cancer"

This is just incorrect, people with chronic pancreatitis have massively increased risk of developing pancreatic cancer (16x IIRC). There also seems to be a hereditary factor.

Czech healthcare system, in fact, has a limited pancreatic cancer screening program since 2024, for people who were identified as high-risk.

https://www.cgs-cls.cz/screening/program-vyhledavani-rakovin...

[dgroshev]

Prolonging the expected lifetime by several years nontrivially improves chances of surviving until better drugs are found, and ultimately long term survival. Our ability to cure cancers is not constant, we're getting better at it every day.

[holowoodman]

Even so. Current first-line treatment for pancreatic cancer is surgery, because chemo doesn't really help a lot. Chemo alone is useless in this case. So any kind of treatment that does have a hope of treating anything involves removing the pancreas.

Take those 99% false positives. If you just remove the pancreas from everyone, you remove 20 years of lifetime through severe diabetes. In terms of lost life expectancy, you killed up to 25 people. Surgery complications might kill one more. In all, totally not worth it, because even if you manage to save everyone of those 1% true positives, you still killed more than 20 (statistical) people.

And the detection rate might be increased by more testing. But it needs to be a whole lot more, and it won't help. Usually pancreatic cancer is detected in stage 3 or 4, when it becomes symptomatic, 5 year survival rate below 10% (let's make it 5% for easier maths). The progression from stage 1 to stage 3 takes less than a year if untreated. So you would need to test everyone every 6 months to get detections into the stage 1 and stage 2 cases, that are more treatable. Let's assume you get everyone down to stage 1, with a survival rate of roughly 50% at 5 years, 15% at 10 years. We get a miracle cure developed after 10 years where everyone who is treated survives. So basically we get those 15% 10-year-survivors all to survive to their normal life expectancy (minus 20 because no more pancreas). Averaging they get an extra 10 years each.

Pancreatic cancer is diagnosed in 0.025% of the population each year. In the US at 300Mio., thats 750k in 10 years. With our theoretical miracle cure after 10 years for 15%, that is a gain of 1.125Mio years lifetime. A 1 hour time needed for testing per each of 300Mio people twice a year for 10 years already wastes 685k years of lifetime, so half the gain already. That calculation is already in "not worth it" territory if the waiting time for the blood-draw appointment is increased. That calculation is already off if you calculate the additional strain on the healthcare system, and the additional deaths that will cause.

[thaumasiotes]

> If the stats were as good as the hyperbole in the article, it would clearly state the only 2 metrics that really matter: predictive value positive (what's the actual probability that you really have cancer if you test positive) and predictive value negative (what's the actual probability that you're cancer free if you test negative). As tptacek points out, these metrics don't just depend on the sensitivity and specificity of the test

This is a bizarre thing to say in response to... a clear statement of the positive and negative predictive value. PPV is 62% and NPV is "over 99%".

Your calculations don't appear to have any connection to your criticism. You're trying to back into sensitivity ("the test missed the majority of people who had cancer") from reported PPV and NPV, while complaining that sensitivity is misleading and honest reporting would have stated the PPV and NPV.

[dv_dt]

so possibly saving lives and late stage cancer care level medical expenses 2/3 of positive results vs fear and lighter medical care 1/3 of the time. is this not a win?