[gwern]

This article is a confused mess. It's something of a Gish gallop in conflating all the different issues they could come up with, while leaving out all the necessary vocabulary (C-f "Bayes" "posterior" "decision theory" [Phrase not found]) making it almost impossible to consider each issue in adequate detail.

It mixes up poor communication (reporting false-positive/negative rates as if posterior probabilities, & exaggerated confidence thereof), arbitrary-seeming decision thresholds (but their hyperventilating over '85% wrong' notwithstanding, many are probably too conservative, if anything, given how devastating many of these problems are, there should be more false positives to trigger additional testing, not less), costs of testing (sure why not but little is presented), tests which they claim just bad and uninformative (developed based on far too little _n_, certainly possible), implicit calls for the FDA to Do Something and ban the tests (not an iota of cost-benefit considered nor any self-reflection about whether we want the FDA involved in anything at all these days)... Sometimes in the same paragraph.

Plenty of valid stuff could be written about each issue, but they'd have to be at least 4 different articles of equivalent length to shed more light than heat.

[hn_throwaway_99]

> implicit calls for the FDA to Do Something and ban the tests (not an iota of cost-benefit considered nor any self-reflection about whether we want the FDA involved in anything at all these days)...

This is true in so many areas of journalism but lately seems especially egregious in the NYT. And I don't really blame them, as the incentives for any individual reporter are just too great - having the government make a major policy change based on your article is basically the brass ring for an investigative reporter.

I basically can only use these types of articles as a jumping off point for my own research, as I usually find the moralizing conclusion the article comes to as unsupported.

[nohuck13]

"the incentives for any individual reporter are just too great - having the government make a major policy change based on your article is basically the brass ring for an investigative reporter"

Yep, this is the framing I came here looking for.

Investigative journalists live in the same asymmetrically-incentivized world as social science researchers. If the reporter had looked into the phenomenon and concluded "yeah, boring technical logic pretty much works as expected here" then there's no story.

[nohuck13]

I wonder if the NYT editorial staff receives zillions of these pitches from their reporters purporting to reveal nefarious phenomena, and most of them turn out benign. It's fun to imagine that the NYT editorship is actually exceedingly good at detecting these outrage-false-positives before publication, but the base rate of outrage-true-positives is just so low that you have to expect some to make it to publication.

I'd like to imagine there's an investigative-journalism-editor-news somewhere, and they're discussing this discussion saying "bah, these clowns are making sweeping generalizations about editorial standards based on only a false positive; this is totally specious with no mention of the prior distribution or sensitivity vs specificity trade-offs"

[bscphil]

They even missed "base rate", which is the way I usually see this explained to ordinary people without stats backgrounds. Really disappointing.

[SpicyLemonZest]

They don't use that specific term, but the Down syndrome infographic does a pretty solid job at explaining the base rate issue.

[creata]

> implicit calls for the FDA to Do Something and ban the tests

Not that you're necessarily wrong, but how did you get that from the article? It didn't seem to me like they wanted a ban.

[mcguire]

So you are saying the testing companies in the article aren't fraudulently claiming much more effective tests than they are providing?

[jrockway]

Specificity and sensitivity are two dimensions that you can measure tests in. You can claim your test is 99% accurate if you mean that "if the test says you don't have the disease, there is a 99% chance that you don't have the disease". That same test can still be 85% wrong if it says you DO have the disease, though.

I doubt that hyping one side of this equation is fraud. Pushing the error in this direction seems like a good idea, anyway. If you have some weird illness, and the test comes back as a false positive, at least you'll continue to explore that possibility for a while. If it comes back as a false negative, then you'll spend a ton of time exploring alternatives which will be true negatives. Probably infuriating.

https://en.wikipedia.org/wiki/Sensitivity_and_specificity

[mcguire]

Here's the fliers mentioned in the article:

https://www.harmonytest.com/content/dam/RMS/harmonytest/glob...

https://web.archive.org/web/20211116203541/https://myriadwom...

https://images.health.questdiagnostics.com/Web/QuestDiagnost...

I'd appreciate it if you could point out where any of them walk the potential customer through sensitivity, specificity, and the fact that if they test positive, there is an 80-90% chance that their will not be affected. I can't seem to find any of that.

[in_cahoots]

When I got my test results, they were clear that the odds of having a disorder were (for example) 1/144, even with a ‘positive’ result. This was through Natera. The problem is that this information is sent directly to the provider in most cases, so parents are left interpreting someone else’s interpretation of statistics. My midwife specifically told me that the test isn’t often wrong, even though the actual odds were there in the fine print.