[neuro_image3]

There are several key points that get left out in AI radiology conversations such as this one:

1) Mammograms are not interpreted in a vacuum. In fact mammograms are usually the first in a long line of tests before a breast cancer or other diagnosis is ultimately made. In fact, it's probably more accurate to refer to mammography as a screening exam for which patients need a biopsy rather than a diagnostic test for cancer (there are rare exceptions, but overall this point holds).

2) Speaking frankly as a radiologist myself, tests like mammograms aren't even that good in terms of overall diagnosis. Thats why ultrasound, tomosynthesis and MRI are often used as supporting evidence and/or alternative exams.

3) There is controversy over the overall utility of mammograms, particularly in the screening context. Radiologists more than anyone would like the sensitivity and specificity of these studies to be higher.

It strikes me that the people that push these "radiology is ripe for disruption" or "AI outperforms radiologists" hyperbolic arguments are clearly people that have never seen the inside of a clinic. I'm sure they love this rhetoric though when pitching to VCs or sitting around the conference table coming up with 'breakthrough ideas' to turn into power-points for the other administrators.

[1_over_n]

We are working in the breast cancer space now looking at breast cancer and ultrasound (not just from a screening / diagnostic perspective - also treatment planning for medical oncologists and treatment response planning).

We don't use deep learning - we use Biophysical models. We hate using the term "AI". This is a very challenging discipline to explain to VCs.

Also, speaking to point 2 here - the "value" of building tools for ultrasound is often dismissed by VCs because "ultrasound isn't used for screening or diagnosis". This is an insane perspective from our position when we are practically based within hospitals, collaborating strongly with radiologists and medical oncologists who work with ultrasound on a daily basis.

We are very embedded within the hospital and look to understand the clinicians workflow and decision making processes first, as well as understanding what's possible given the hurdles involved in data access (which can still be tricky even when you are through IRBs and ethics).

We have found that telling VCs the reality about working with hospitals and doctors can often limit their excitement about your company prospects. Our success to date has largely been as a result of doctors and hospitals who believe in us, see the value in what we are doing. They have put time and effort into collaborating because they are impressed with what we have been able to do results wise by bootstrapping as a small team, rather than as a VC funded shiny startup.

In a weird way i would say that at it's best times medtech can be one of the "purest" industries to work in. By this i mean ultimately your technology works or it doesnt (at least from the medical communities perspective - again VCs are a different story). There are obviously exceptions to this (Theranos anyone) and there are issues around the 510K process but on the whole there is a big price to pay for making unsubstantiated claims (say compared to aspirational lifestyle marketing).

[jlg23]

> 1) Mammograms are not interpreted in a vacuum. In fact mammograms are usually the first in a long line of tests before a breast cancer or other diagnosis is ultimately made.

The paper specifically talks about mammography, it does not claim to replace a complete diagnosis.

> 2) Speaking frankly as a radiologist myself, tests like mammograms aren't even that good in terms of overall diagnosis. Thats why ultrasound, tomosynthesis and MRI are often used as supporting evidence and/or alternative exams.

From the abstract: "2) successfully extends to digital breast tomosynthesis"

> 3) There is controversy over the overall utility of mammograms, particularly in the screening context.

> It strikes me that the people that push these "radiology is ripe for disruption" [...]

The paper, which I just skimmed over, does not read hyperbolic, for that we'll have to wait for popsci journalists.

OTOH, if one leaves the 1st world context, any type of successful diagnosis automation in medicine is a blessing for areas where you simply don't have enough trained medical staff.

[neuro_image3]

I wasn't just commenting on the abstract presented. I was commenting on the comments I see here, as well as comments I see related to similar papers all the time.

I also interact with AI/ML researchers all the time. Most of them are typically some combination of: 1. Poorly informed about the appropriate context and utility of medical imaging. 2. Trying as hard as they can to push AI/ML as the most important technology in medicine today. 3. Pursuing a very task-specific project which they claim is massively generalizable in some (incorrect) way.

[kilburn]

There's one point that often comes up when I chat with my MD friends: All of them agree that more information is not strictly better while diagnosing. In fact, most support that unneeded information is actually worse because it confounds the issue.

My engineering mind just can't come up to terms with this. Why wouldn't you collect all information you possibly can? You can always ignore irrelevant data you have, but you cannot consider data that you don't have!

The closest I've been to rationalizing this: diagnosing is a stochastic process so complex (and with the search space so large) that the random noise in extra data is likely to point you towards wrong directions. Plus you can always collect more data afterwards if your initial diagnosis turns out to be wrong. This is of course very simplified, but it makes sense.

However, I just can't turn off my inner voice from screaming "more data is always better". I guess that's why I'm not an MD :)

[inimino]

> You can always ignore irrelevant data

Everything we know about human psychology says you can't.

[1e-9]

> Why wouldn't you collect all information you possibly can?

From a decision theoretical viewpoint, you would certainly want all the information you could get. For humans running a business in today's medicolegal environment, it's a very different set of issues:

1) Collecting information costs time and money.

2) Making good decisions requires the most precious resource of all, which is doctor brain-time. There isn't enough of it to spend on information with little probability of benefit.

3) If you get sued for malpractice, the unneeded data you collected probably would not have helped the patient, but it could help the attorneys arguing that you missed something. Juries struggle to understand the cost of false positives.

Even though there are valid issues here, doctors don't always make the right tradeoff in this regard. Oftentimes, I think it is more an issue of lack of training or experience that leads a doctor to consider a test to be unneeded. In the case of mammography, if doctors spend too much time doing screening and not enough time doing diagnosis, their screening performance degrades, which I think is due to a lack of feedback on their decision making[1].

[1] https://www.ncbi.nlm.nih.gov/pubmed/21343539

[DanBC]

It's pretty hard to ignore that extra information though.

You run a screening company. You take people at high risk of lung cancer -- people who smoke a lot and have smoked a lot for many years -- and you provide low dose CT scans of their lungs.

Bob comes in. You scan his lungs and you find spots.

What do you do now?

You're probably going to start providing treatment to Bob. Will this help Bob live longer? Will it improve his quality of life? It might not.

https://blogs.bmj.com/bmjebmspotlight/2019/02/15/understandi...

[TeMPOraL]

> What do you do now?

Hopefully get other tests done, to confirm diagnosis.

I'm with GP here. I can't understand this attitude either. Having more information should never make you more wrong. This holds for uncertain information, because uncertainty can be quantified and tracked (if you're not doing this, then you're doing voodoo, not science).

I can see two reasons why you wouldn't want to gather more information in medical context. One, many tests carry risk to patient's health and well-being, so there's no point of doing them if that risk outweighs the expected value of evidence gathered. Two, I suspect that gathering information also gathers legal obligations and risks to doctors.

[DanBC]

> Hopefully get other tests done, to confirm diagnosis

Those other tests involve things like "needle biopsy" -- they shove a needle through your chest into your lung into the suspect tissue to get a sample. This carries risk. We can justify that risk if it saves life. But this is the problem with screening -- often it doesn't save life (of course, it depends on the type of screening).

https://www.radiologyinfo.org/en/info.cfm?pg=nlungbiop

> Having more information should never make you more wrong

But you can see how having lots of low-quality information could make someone more wrong -- these are not clear signals, because if they were it wouldn't be a problem. These are almost noise. We're taking data from a large population ("4 in 100 people with this result have this disease") and trying to apply it to the individual, and when we try to get more information we subject this person to more radiation in scans or invasive procedures or both. We increasing the risk, but not necessarily saving life.

> there's no point of doing them if that risk outweighs the expected value of evidence gathered

Yes, this is exactly the balance that doctors are making. They're looking at all cause mortality and seeing if life is saved.

[tuberelay]

MD here.

In the short term, more information might cause harm because doctors are risk averse & scared of lawsuits and err on overbiopsy/overtreat, and many of our treatments aren't as good as we think they are, and all of this makes patient anxious.

In the long term, turning the information firehose on full blast means we can work out which incidental findings are best ignored or pursued and overall more data will help us.

The problem is that it is unethical to do #2 in the short term even if it is the long term ethical thing to do.

[jlg23]

I also interact with medical doctors all the time: Most of them are typically some combination of: 1. Poorly informed about applicability or claims thereof of CS methods. 2. Trying as hard as they can to push the image of "the human doctor always knows best". 3. Pursuing university degree and then work in a very narrowly defined field without much relevant further education/updates believing their now 50 year old knowledge is set in stone.

I completely get your attitude, I think I agree with you overall and if I was not this lazy I could comb through my bookmarks and find the studies supporting what you said.

But I was just responding to your comment in the context of the paper linked. Which, at least when skimming over it, does not read like what you (IMHO, rightfully) criticize in the broader debate.

And yes, read the first paragraph as a tongue-in-cheek response, we both know that overgeneralizations don't help any debate ;)

[1e-9]

Your comments do not seem to be addressing this particular study, but rather seem to be directed at the plethora of poorly-designed/over-hyped ML papers that are published on a regular basis. While that is an issue, this particular paper made no claims about disruption and it had nothing to do with diagnostic performance. It was a screening study that appears to make sound comparisons to the performance of five reasonably-qualified radiologists on screening images.

[wdevanny]

I think their comment is a very needed bucket of ice water on the multitude of other comments in this thread that are making claims about disruption and diagnostic performance.

[1e-9]

Can you point to a particular comment in this thread making a claim about disruption or diagnostic performance?

[neuro_image3]

The next main root comment. 'Of the major specialties, it seams that radiology is the most in danger of disruption'.

I wonder how long people think about these sorts of claims before posting them.

Do they really think an AI is more likely to appropriately interpret an MRI scan (and all the anatomic, physics and pathophysiological data therein contained) in the context of a specific clinical work-up more easily than triage patients the way a family practitioner or ER doctor does?

[TeMPOraL]

I'm as skeptical about today's "AI" as you can get, but FWIW, medical community seems concerned about it. Going by what my acquaintance fresh out of medical school keeps telling me (and what the doctors chambers' publications seem to be saying), doctors are worried about the impact of AI on their jobs, and they believe the (IMO vastly exaggerated) claims of effectiveness of upcoming solutions. And radiology does seem to be at the forefront of this - the question whether it's even worth it to start specializing in radiology today is one seriously considered by graduates.

[jeffreygoesto]

AI is not "interpret"ing. It is merely using a fitted distribution. Just because there are many degrees of freedom fitted does not make it magic.

[1e-9]

There was no claim in that comment. "It seems" implies a personal opinion of the poster.

[ggm]

I wish there was a way to repost this to all most every medical breakthrough ml story.

[cameronfraser]

It could be useful as a tool to help a radiologist do their job better though. I think many of these techniques described in ML papers will be used to enable people to be better at their jobs rather than replace them. At least until there is AGI at least.

[neuro_image3]

I wouldn't dispute this (if they finally put something together that isn't horrendously cumbersome, time-consuming and hard to use) but this doesn't justify the 'AI is about to replace radiology' crap I seam to see every time some academic group publishes an AI/ML paper.

[cameronfraser]

I don't think this paper makes that claim, you might be thinking of media interpretations of papers which is usually the one making bogus claims. It says that it outperforms 5 out of 5 people, but that is in this specific context. They aren't necessarily claims meant to be generalized that much.

[neuro_image3]

I see that type of hyperbolic claim several times in the comments.

I wasn't just commenting on the abstract presented. I was commenting on the comments I see here, as well as comments I see related to similar papers all the time.

I also interact with AI/ML researchers all the time. Most of them are typically some combination of: 1. Poorly informed about the appropriate context and utility of medical imaging. 2. Trying as hard as they can to push AI/ML as the most important technology in medicine today. 3. Pursuing a very task-specific project which they claim is massively generalizable in some (incorrect) way.

[StreamBright]

Usually they do not understand that your workflow if not revolving around pattern recognition on pictures. The best use of ML in pattern recognition for radiology is ordering images. You would get the the images sorted by the likeliness of having something unusual.

[StreamBright]

What do you think how far AGI is?

[cameronfraser]

a long long time

[StreamBright]

I guess over 100 years at the very least.

[StreamBright]

The only thing I see as potential area of improvement (i do not like the word disruption) in radiology is workflow optimization to reduce the time you spend on administrative tasks. I would be interested to hear your opinion on this.

[neuro_image3]

Absolutely, that's an area where various software innovations could be extremely helpful.

In fact, the area of medicine most in need of 'disruption' imho is healthcare enterprise software. Doctors are literally killing themselves because the interfaces they have to deal with on a daily basis are so appallingly poor.

Of course, the solution is less technological than political as it wouldn't take much to come up with better software than the legacy alternatives but you'd have a very hard time getting past the entrenched relationship interests of crony bureaucrats that run hospital administrations.

[ekianjo]

If I remember correctly for some women a mammograph can be completely unreadable while much clearer for other's. That seems to make it a very poor method for diagnosis, despite it being recommended almost everywhere.

[llsf]

Don't you think though that it would help you? I see this ML treatment of the images as a resolution improvement. It would help doctors to see better, maybe to not miss the detail that could be missed.