[frankc]

But it sounds like with AI doctors did better overall, or that is how I read the first couple of lines. If that is true, I don't really see a problem here. Compilers have eroded my ability to write assembly, that is true. If compilers went away, I would get back up to speed in a few weeks.

[jackvalentine]

The compiler example is very helpful, thanks for posting it.

My follow up question is now “if junior doctors are exposed to AI through their training is doctor + AI still better overall?” e.g. do doctors need to train their ‘eye’ without using AI tools to benefit from them.

[benterix]

Yeah everybody compares traditional programming to using assembler now. This analogy would be great except that the current generation of LLMs is still quite unreliable and a large part of what it generates is a mess.

In a perfect world, there would be no problem - those of us who enjoy the experience would vibe-code whatever they need, and the rest of us would develop our skills the way we want. Unfortunately, there is a group of CXOs, such as the ones at Github, who know better and force the only right way down their employees throats.

[jackvalentine]

These radiography AIs aren’t LLMs AFAIK.

[joe_the_user]

Any of these neural networks is going to have the same basic qualities - lack of explainability and unreliability (even if they have many good qualities, are "fairly" reliable etc, they do fail and we don't exactly when/why they fail).

[frankc]

I think that is a good question and we don't really know yet. I think we are going to have to overhaul a lot of how we educate people. Even if all AI progress stops today, there is still a massive shift in how many professions operate that is incoming.

[jackvalentine]

> I think we are going to have to overhaul a lot of how we educate people.

I agree.

I work in healthcare and if you take a tech view of all the data there are a lot of really low hanging fruit to pick to make things more standardised and efficient. One example is extracting data from patient records for clinical registries.

We are trying to automate that as much as possible but I have the nagging sense that we’re now depriving junior doctors of the opportunity to look over hundreds of records about patients treated for X to find the data and ‘get a feel’ for it. Do we now have to make sure we’re explicitly teaching something since it’s not implicitly being done anymore? Or was it a valueless exercise.

The assumptions that we make about training on the job are all very chesterton’s fence really.

[benterix]

I think this is has already started a while ago with data science. While it might not be as fashionable as before, it's really at the core of many of these jobs where various form of machine learning or generative AI are being used.

[dweinus]

Except the paper doesn't say that the doctors + AI performed better than doctors pre-AI. It is well documented that people will trust and depend on AI, even if it is not better. It is not clear in the paper, but possible that this is just lose-lose.

Paper link: https://www.thelancet.com/journals/langas/article/PIIS2468-1...

[mcbrit]

Line: The ADR of standard colonoscopy decreased significantly from 28·4% (226 of 795) before to 22·4% (145 of 648) after exposure to AI

Supprt: Statistically speaking, on average, for each 1% increase in ADR, there is a 3% decrease in the risk of CRC. (colorectal cancer)

My objection is all the decimal points without error bars. Freshman physics majors are beat on for not including reasonable error estimates during labs, which massively overstates how certain they should be; sophomores and juniors are beat on for propogating errors in dumb ways that massively understates how certain they should be.

This article is up strolls rando doctor (granted: with more certs than I will ever have) with a bunch of decimal points. One decimal point, but that still looks dumb to me. What is the precision of your measuring device? Do you have a model for your measuring device? Are you quite sure that your study, given error bars, which you don't even acknowledge the existence of, don't cancel out the study?

[dweinus]

Either way, to be clear: the 28.4% -> 22.4% is human performance vs human performance (before and after "exposure to AI"). There are no numbers provided on accuracy with the use of AI.

[itsdrewmiller]

That is paywalled. From another article on the topic (https://bioengineer.org/study-suggests-routine-ai-use-in-col...)

> While short-term data from randomized controlled trials consistently demonstrate that AI assistance improves detection outcomes...

[joe_the_user]

The compiler analogy is seductive but problematic imo.

A compiler can be fixed thing that does a fixed task. A cancer recognizer is something like a snapshot of people's image-recognition process during a period of time. These are judgement that can't be turned into set algorithms directly.

There was a discussion a while about how face recognition trained with Internet images has trouble with security camera footage 'cause the security camera doesn't certain images.

It sounds weird to say that what cancer looks like drifts over time but I'm pretty sure it's actually true. Demographics change, the genes of even a stable group change over the generations, exactly how a nurse centers bodies, etc. change over time and all these changes can add to the AI judgement snapshot being out of date after some period. If the doctors whose judgements created the snapshots no longer have the original (subtle) skill then you have a problem (unlike a compiler whose literal operations remain constant and where updating involves fairly certain judgements).

[Ukv]

> change over time and all these changes can add to the AI judgement snapshot being out of date after some period

The detector can be fine-tuned over time to improve accuracy and account for distribution shift. We should still have access to the same signals (did this turn out to be a tumor?) that doctors would use to update their own judgement.

[lurking_swe]

if the hospital IT system is temporarily down, i certainly expect my doctors to still be able to do their job. So it is a (small) problem that needs addressing.

Perhaps a monthly session to practice their skills would be useful? So they don’t atrophe…

[sothatsit]

If the AI systems allow my doctor to make more accurate diagnoses, then I wouldn't want a diagnosis done without them.

Instead, I would hope that we can engineer around the downtime. Diagnosis is not as time-critical as other medical procedures, so if a system is down temporarily they can probably wait a short amount of time. And if a system is down for longer than that, then patients could be sent to another hospital.

That said, there might be other benefits to keeping doctors' skills up. For example, I think glitches in the diagnosis system could be picked up by doctors if they are double-checking the results. But if they are relying on the AI system exclusively, then unusual cases or glitches could result in bad diagnoses that could otherwise have been avoided.

[randcraw]

Or, each time the software is updated or replaced, the human in the loop will be unable to... be in the loop. Patients will be entirely dependent on software vendor corporations who will claim that their app is always correct and trustworthy. But what human (or other external unbiased trustworthy authority) will be available to check its work?

[stavros]

Do you expect doctors to be able to image your insides if the X-ray machine is down too?

[lurking_swe]

An x ray machine can be used “locally” without uploading the images into the IT system. So i don’t understand the question. If it was designed to be cloud only then that would be horrendous design (IMO).

The x ray machine would still work, it’s connected directly to a PC. A doctor can look at the image on the computer without asking some fancy cloud AI.

A power outage on the other hand is a true worst case scenario but that’s different.

[stavros]

I'm not talking about the IT system, I'm talking about when the X-ray machine breaks, same as how we're talking about when the colonoscopy diagnosis machine breaks.

[fzeroracer]

How often do you think the x-ray machine breaks vs how often software shits the bed?

Like one of the biggest complaints I've heard around hospital IT systems is how brittle they are because there are a million different vendors tied to each component. Every new system we add makes it more brittle.

[d3vnull]

Seems like a pretty easy fix in treating the system that runs the cancer detection algorithm as an hospital machine and not as part of the IT system. It can be an airgapped system that runs just the needed software in a a controlled configuration.

This is not new, lots of mission critical software systems are run like this.

[yread]

If the hospital IT system is down, hospital stops. That's why hospitals spend resources so that it doesn't happen

[oliwarner]

A lot of imaging and note-taking is digital now, so AI aside, they probably won't meet your expectations.

[akoboldfrying]

> if the hospital IT system is temporarily down

I think we have to treat the algorithm as a medical tool here, whose maintenance will be prioritised as such. So your premise is similar to "If all the scalpels break...".

[tylerrobinson]

I’m think you might agree, though, that the likelihood of one premise is significantly greater than the other!

[akoboldfrying]

Sure. "MRI machine" would have been a better metaphor than "scalpel".

[azemetre]

Which is easier to build resilient systems for: the one where you have a few dozen extra scalpels in a storage closet or the one that requirements offsite backups, separate generators, constant maintenance?

The premise is absolutely not the same.

[akoboldfrying]

s/scalpel/MRI machine/. How about now?

[azemetre]

Sounds like a great system that benefits from having lots of money. IDK how such a thing can last in rural areas where there may be one single MRI machine to use in a 100 mile radius.

[add-sub-mul-div]

"People will practice their skills" is the new "drivers will keep their attention on the road and remain ready to quickly take the wheel from the autonomous driver in an emergency."

[xorbax]

It's like research. People had encyclopedias. If they wanted to know real, deep information about subjects they'd have to specifically spend effort seeking and finding books or papers about that specific subject (which are typically just distilled papers in a far wider range and number than an encyclopedia would be)

Then we could just go Google it, and/or skim the Wikipedia page. If you wanted more details you could follow references - which just made it easier to do the first point.

Now skills themselves will be subject to the same generalizing phenomenon as finding information.

We have not seen information-finding become better as technology has advanced. More people are able to become barely-capable regarding many topic, and this has caused a lot of fragmentation, and a general lowering of general knowledge with regard to information.

The overall degradation that happened with politics and public information will now be generalized to anything that AI can be applied to.

You race your MG? Hey my exoskeleton has a circuit racer blob we should go this weekend. You like to paint? I got this Bougereau app I'll paint some stuff for you. You're a physicist? The font for chalk writing just released so maybe we can work on the grand unified theory sometime, you say you part and I can query the LLM and correct your mistakes

[salawat]

>Then we could just go Google it, and/or skim the Wikipedia page. If you wanted more details you could follow references - which just made it easier to do the first point.

Except at this point, market forces and going whole hog on neural networking and such instead of sticking with just reflective, impartial indexing of the digital medium has made it nigh impossible for technological aid to actually improve your ability to find niche things. Search Engine Optimization, plus the interests in shaping narratives, have made searchability take a plunge. Right now the unpolluted index may as well be a WMD for how hard it is to find/keep operating one.

[seanmcdirmid]

This already happened in aviation a long time ago, they have to do things to keep the pilots paying attention and not falling asleep on a long haul where the auto pilot is doing most of the work. It isn't clear at what point it will just be safer to not have pilots if automated systems are able to tackle exceptions as well as take offs and landings well enough.

[accoil]

Does cancer progress that fast?

[Forgeties79]

The stakes of a colonoscopy are typically way, way higher than your typical assembly projects.

[akoboldfrying]

Think of it as a medical device, like an MRI machine. Should we have workarounds for when the MRI machine is down? I think we are better off allocating budget to keeping the MRI machine maintained and running, and assuming that as the normal state -- and likewise for this.

[jghn]

In many cases these software tools are literally classified as medical devices by the FDA with all of the regulatory compliance that comes with it

[Forgeties79]

As the other person said - an MRI has way more oversight/rules surrounding it to insure it’s functioning properly and that people are held accountable when it isn’t. The same can’t be said here.

They aren’t parallel situations and you can’t cleanly graft these requirements over.

[xorbax]

I think the same thing about meals.

[WhyOhWhyQ]

You would get back up to speed in a few weeks. The guy who comes after you and never had formative years writing assembly would never get to the level you were at.

[frankc]

Perhaps, but I don't think we should optimize for scenario of going back before these tools existed. Of course you need the medical equivalent of BCP, but it's understood that BCP doesn't imply you must maintain the same capacities, just that you can function until you get your systems back online.

To continue to torture analogies, and be borderline flippant, almost no one can work an abacus like old the masters. And I don't think it's worth worrying about. There is an opportunity cost in maintaining those abilities.

[nfnriri8]

Back up to speed.

Who gets the next generation "up to speed" if the teachers are always forgetting?