[halpert]

How did this article, written by someone who clearly lacks an understanding of basic statistics, make it into the Upshot? They try to make it seem like the test is wrong 85% of the time, but that's not necessarily the case. All we know from the article is that 85 / 100 positive results are false positives, which means the test could actually be quite accurate. If the test correctly identifies 100% of real cases, then that sounds like an excellent test. Just as an example, if 1/4000 people have the disease, and the test identifies 100% of these cases, then around 0.14% of test takers will get a false positive.

[SpicyLemonZest]

Their infographics convince me that they understand the statistics. But one of the key issues here is that the statistics are radically counterintuitive in a way that most people don't understand - the patients, the testing companies, and even some medical staff all incorrectly believe that a positive test for a rare condition means you probably have the condition.

[halpert]

Their graphics say the tests are “84% wrong.” Do you really feel that’s an accurate description? That doesn’t feel like an accurate description to me, and their usage of “wrong” in this context highlights that they don’t understand the distinction and importance of true positives, false positives, true negatives, and false negatives when measuring accuracy.

[isoprophlex]

Going through something like this is very VERY stressful. When you get a negative you immediately forget about it. When you get a positive you die inside. Speaking from experience here.

84% wrong sounds, to me, as an accurate description. Experiencing this from the inside out, only the false/true positive ratio matters. (Given sufficiently low false negative rates, of course)

84% of people whose world is turned upside down are actually getting a wrong diagnosis.

[andreilys]

You’re talking about precision (true positive / true positive + false negative) but that’s only one part of the story.

There is a real human cost to having a child born with a rare genetic disease (and I would argue is immensely more stressful). You can easily adjust the sensitivity to the test but at the cost of detecting actual true positive cases. The correct response to receiving a positive is to do another test to ensure it’s not a false positive.

To say 84% wrong is clickbait and used to elicit a legislative response (FDA regulation), which will help the reporters career.

The actual ratio to tell if something is “wrong” is accuracy (True positive + true negative) / (true positive + true negative + false positive + false negative)

[mnw21cam]

No, precision is true positive / (true positive + false positive).

Your first equation is sensitivity.

[fshbbdssbbgdd]

If you get a negative result and then your child is born with the condition, you won’t forget quickly either.

[SpicyLemonZest]

I really feel it's an accurate description. If you get a positive result on the test, there's a 16% chance your fetus has a 1p36 deletion and an 84% chance they don't.

[halpert]

As you said “if you get a positive result”. It’s true, if you ignore the 99.9% of the time the test is correct (true negative result), then you can say the test is 84% wrong.

[SpicyLemonZest]

84% of people who got a positive test result will end up telling their family "it's OK, the first test was wrong, my baby doesn't have a 1p36 deletion after all". The 99.9% of other people who got true negatives are important from a test design perspective, because specificity is closer to the actual levers you can pull on, but it's not super relevant to the decisionmaking process of someone who gets a positive result.

[andreilys]

Ignoring all the true and false negatives which themselves are markers of how accurate the test is.

16% precision is the correct statement, saying the test is wrong 84% of the time implies that those getting negative results might actually have positive results.

[robbedpeter]

He framed his statement correctly, limiting his observation to the condition that the test returned a positive result. Saying that 84% of positive results are false is correct if only 16% are true. You'd need to know false negative rates and base occurrence rates (modified by whatever other factors are unique to your situation) to inform the nature of information you get by performing the test.

[ellisv]

I disagree. It is clear from the title, “When They Warn of Rare Disorders, These Prenatal Tests Are Usually Wrong”, and the lead that they’re focusing on false positives.

[halpert]

It's true they are focusing on false positives, but the authors are using the ratio of false positives to true positives to paint a picture that the tests are inaccurate, when in reality the tests are accurate. What this article is looking at is called the "sensitivity" of a test: https://en.wikipedia.org/wiki/Sensitivity_and_specificity

[mnw21cam]

No, the article isn't talking about sensitivity. We don't actually know what the sensitivity is from the data the article gives us. We are told that lots of people were screened and a small number had a positive result, of which a proportion were actually positive. You can't calculate sensitivity from that because you don't know how many actually positive cases were missed.

This article is talking about precision, which is the proportion of positive results that are true. And it's okay for precision to be awful, especially when the condition is so rare. But it's only okay if the result is communicated alongside a statement saying what the precision is, which it seems these were not.

[halpert]

Yes you are correct.

[adjkant]

While the author may not be well versed or focusing on the stats side, you're missing the human side here I think.

> the tests are inaccurate, when in reality the tests are accurate

If the test make someone consider terminating a pregnancy or even considering it, that's a lot of pain. So for that human, the test is failing its purpose potentially, depending on the value calculation of terminating a viable pregnancy vs the severity of the issue if it comes to term.

For a human, accuracy as you defined it means little to nothing. Usefulness and helpfulness are far better metrics, and such a high false positive rate is clearly causing issues in respect to those, which is what the article is highlighting.

[andreilys]

Usefulness and helpfulness are far better metrics, and such a high false positive rate is clearly causing issues in respect to those

How exactly do you plan on codifying usefulness and helpfulness?

A high false positive rate is not necessarily a bad thing and may instead be the catalyst for additional tests to confirm the first one. The tests accuracy may actually be 100%, which is great because it avoids a child being born with a fatal genetic disease. Would you prefer a high false negative rate that misses these diseases instead?

[halpert]

Or maybe you’re missing the human side of having a child born with a serious genetic defect?

[mcguire]

Is it better to terminate 85 pregnancies which do not have a serious defect in order to catch 15 which do? At what point is it not better to terminate 100% of pregnancies?

[loeg]

> Is it better to terminate 85 pregnancies which do not have a serious defect in order to catch 15 which do?

Yes, it’s absolutely better to do that. Of course, the actual ratio is much better than that because we do follow-up tests after the screen.

[paulryanrogers]

> At what point is it not better to terminate 100% of pregnancies?

Everyone should decide for themselves. Having seen the long term consequences I would rather err on the side of caution, even if it were difficult to become pregnant.

Such diseases are often incurable and significantly degrade the quality of life of not only the person to be born but the whole immediate family. At least in the US the there isn't enough social safety net or support too offset the crushing costs.

[ramraj07]

Did they use the word accurate? You used the word accurate and then you yourself are going on a tirade about how that’s not correct?

It’s clear the article is talking about why sensitivity is important in layman’s terms and while it could use better writing it’s a real problem in diagnostics. This is why you don’t ask men to take a pregnancy test to check for prostrate cancer. It is accurate but not sensitive.

[halpert]

They used the word “wrong”. Whether or not they used wrong to mean inaccurate, or wrong to mean not sensitive is up to the reader.

[wizee]

The issue is that the tests portray themselves as being accurate (in the sense of low false positive rates), and portray the result as “your baby has XYZ rare syndrome” instead of “your baby has a 15% change of having XYZ rare syndrome”. If the test providers stated the false positive rate for their results more clearly, parents would be in a better position to make informed decisions.

[rflrob]

The larger issue as I see it is that the medical system around these screenings are not well versed in the statistics and able to communicate that to patients. "Eight [patients] said they never received any information about the possibility of a false positive, and five recalled that their doctor treated the test results as definitive." It's hard to know what happened in the room when the doctor spoke with them or what was on those particular patients tests, and that's (one hopes) the worst medical news those people will receive for a long time so listening comprehension is understandably impaired, but there needs to someone available who can help them interpret, even days or weeks later, and these people were let down by the entire system, not just the test manufacturers.

[whakim]

I don't think that it's useful for articles like this to try to educate readers on the way that a precision-recall curve works (and how that differs from the statistical definition of accuracy). Honestly, that would just confuse the vast majority of readers when it's simpler to point out that the tests produce more false positives than they might otherwise expect. Also note that even if we want to be incredibly pedantic, the article never calls the tests "inaccurate" and instead uses a layperson term without a hidden definition ("wrong").

[mcguire]

Would a test that reported 100% positive similarly be "quite accurate"? It would catch all true positives, right?