[andy99]

   99% accuracy in diagnosing benign case

This is meaningless. The only thing that matters in this kind of application is false negative rate at some acceptable false positive rate.

I assume whoever is working on this knows that, so this is mostly a criticism of the article. That said, this is a horrible use of AI.

[Spooky23]

It’s really all about the protocol. AI tends to spot things that doctors don’t, and vice versa. Dermatology is also an area being pillaged by private equity and access is poor for many people.

I lost my wife to melanoma. She noticed a lesion within days of it appearing, and a doctor saw it within 48 hours and felt it was benign. My wife didn’t accept it and had a plastic surgeon remove it and biopsy, then had a margin removed by surgical oncologist, the standard of care at the time. It came back as a brain tumor 4 years later and she was gone in 6 months, even with the incredible advancements today.

So I’d hold the position strongly that anything that improves overall detection rates and access to care is incredibly important and will save lives. Weeks matter with melanoma. Today with immunotherapy Molly would be fine. But if she hadn’t advocated and gotten the original thing removed, it would have cost her 4 important years.

[kennyadam]

Similar story: mom had a melanoma removed from her foot. multiple lymph node biopsies and other tests said it had been successfully removed.

She went back once a year for checks for 4-5 years. It was only when she was called into see an oncologist and told an unrelated x-ray lead them to discover she had stage 4 metastatic melanoma (brain, liver, spine, femur, lungs and i’m sure i’m forgetting something) that we found out that they’d only been giving her visual checkups each year, no PET scans or anything else. The oncologist was shocked that the checks were so basic, mom didn’t know she was supposed to have anything else and she was dead in about 8 weeks.

We were told that the form of melanoma only came back like that in 1% of patients and usually simple visual checkups were enough. I have no idea how true that is.

[Spooky23]

My wife had a similar experience, except it was her shoulder. They took a margin, looked at lymph nodes, and did a nuclear test that traced something that I can’t recall.

Another lesson learned is that if at all possible, go to a national cancer center. Even if for a second opinion analysis. The level of care is different and better than what you find in community oncology or hospital practices.

[tomcam]

So sorry, brother.

[asdefghyk]

I've been told the only way to be sure if skin cancer or not , is a biopsy. I also have been told ... not skin cancer... but Dr decided to send me to a dermatologist. Is thing he said when say it .. is looks like skin cancer ... lets take a biopsy now ... to check . It was skin cancer ... a BCC

[osn9363739]

Dude that sucks mate. I had a melanoma taken off last year. It was dormant (stage 0) but had been there for 10ish years. But reading shit like this reminds me that even though I'm probably fine, all I can do is just live my best life. Hang out with my family. Enjoy the things I enjoy and not think about it too much. (and get my skin checks every 6 months :D)

[kennyadam]

Are your skin checks just visual checks? If so, read my other comment in this thread. It was in UK fyi.

[degamad]

> The only thing that matters in this kind of application is false negative rate at some acceptable false positive rate.

It sounds like they are inverting the scenario here. The question is not "do you have skin cancer?", it's "can you safely go home without seeing a doctor?".

For this new question, we set the acceptable false positive rate to zero (we never want to send a real cancer case home), and determine the false negative rate (we accept that some benign cases will be seen by a doctor).

The reason for the interest in identifying benign cases, rather than trying to identify the positive cases, is that it improves the situation for everybody: benign cases identified by AI are sent home almost immediately, everyone else has a shorter waiting time, so benign false negatives can be assessed more quickly by the doctor and given the all clear, and more time is now available for spending with the real cancer cases.

The numbers they're citing are 7000 cases with 5% real, so 350 real cancer, 6650 benign. If we can accurately say that 6500 of those benign cases are benign without wasting the doctors' time, then we're down to only 500 people needing to see a doctor, which is a huge improvement for everyone.

[greazy]

I went searching for more. The tool is called DERM by Skin Analytics. They have more info on stats:

https://skin-analytics.com/ai-pathways/derm-performance/

A few peer reviewed pubs down the bottom of the page

[potatoman22]

It's trendy to say "it's horrible to use AI for this" without giving specific reasons. Some reasons it could be good to use AI here:

- this can prioritize urgent patients for the severely overworked doctors

- medical error is a leading cause of death, this serves as a second-opinion (97% true-positive rate and 79% true-negative rate)

- it can be used as evidence by a nurse or physician advocating for a patient's treatment

[frereubu]

What do you mean by this being a "horrible" use of AI? (Although as another commenter has mentioned, this should more properly be called ML).

[jampekka]

It's quite easy to correctly classify 100% of benign cases as benign.

[cmcaleer]

If it's so easy, then why do people die from having lesions misdiagnosed as benign?

Even if the success rate of the human eye was in the 99.5%+ range, why not have an extra sanity check from an AI model?

[dijksterhuis]

> If it's so easy, then why do people die from having lesions misdiagnosed as benign?

You're confusing False Negatives with True Negatives. For Non-Benign (Positive) vs. Benign (Negative) classification:

* True Positive Rate (TPR): non-benign classified as non-benign.

* False Positive Rate (FPR): benign misclassified as non-benign.

* True Negative Rate (TNR): benign classified as benign.

* False Negative Rate (FNR): non-benign misclassified as benign.

> It's quite easy to correctly classify 100% of benign cases as benign.

You can engineer a 100% TNR if you just classify everything as the "benign" negative class. The FNR is going to be 100% too, but that doesn't matter -- you correctly classified 100% of benign cases as benign.

> why do people die from having lesions misdiagnosed as benign?

Because the FNR is not 0%. FNR is important. You probably want a decent TPR in there as well. And FPR can be very important too, depending on how life-changing/painful/invasive the treatment for a positive case is!

[OJFord]

Because this non-AI function 'correctly diagnoses 100% benign cases as benign':

    def is_benign(mole: Image) -> bool:
        return True

...but also misdiagnoses 100% malignant cases.

[potatoman22]

To contextualize, I think the tool in this article correctly diagnoses 97% benign cases as benign but misdiagnosis 22% of malignant cases.

[NoPicklez]

I don't think its a horrible use of AI at all, in fact it seems like a fairly good use case.

It's a tool that can be used in amongst the current methods to help detect skin cancer, it shouldn't be used at the only method.

[flir]

Not really. It allows you to order your cases from most-likely to least-likely, and get human eyeballs on the most-likely cases first.

[potatoman22]

This is how most ML in healthcare is used. It's a fantasy to think we can give all patients equal attention, so it often makes sense to prioritize those most at risk. Of course no model is perfect, but ML can be very beneficial when used in the right context (i.e. not like United Healthcare).

[flir]

Yup. The word I should have used is "triage".