[crabbone]

Absolutely this. Also radiologists are usually given notes on patients that accompany whatever image they are reading, and in cases like, eg. ulstrasound often perform the exam themselves. So, they are able to asses presentation, hear patient's complaints, learn the history of the patient etc.

Not to mention that in particularly sick patients problems tend to compound one another and exams are often requested to deal with a particular side of the problem, ignoring, perhaps, the major (but already known and diagnosed) problem etc.

Often times factors specific to a hospital play crucial role: eg. in hospitals for rich (but older) patients it may be common to take chest X-rays in a sited position (s.a. not to discomfort the valuable patients...) whereas in poorer hospitals siting position would indicate some kind of a problem (i.e. the patient couldn't stand for whatever reason).

That's not to say that automatic image reading is worthless: radiologists are, perhaps, one of the most overbooked specialists in any hospital, and are getting even more overbooked because other specialists tend to be afraid to diagnose w/o imaging / are over-reliant on imaging. From talking to someone who worked as a clinical radiologist: most images are never red. So, if an automated system could identify images requiring human attention, that'd be already a huge leap.

[robertlagrant]

You could imagine imprinting into the scan additional info such as "seated preferred" or "seated for pain". There is more encoding that could be done.

[crabbone]

Current "solutions" generally ignore or don't know how to incorporate any textual data that accompanies the image. You are trying to incorporate non-existent data that nobody ever put into any kind of medical system...

Yes, in principle, if people taking the images had infinite time and could foresee what kind of accompanying data will be useful at the analysis time, and then had a convenient and universal format to store that data, and models could select the relevant subsets of features for the problem being investigated... I think you should see where this is going: this isn't going to happen in our lifetime, most likely never.

[jarrelscy]

Developer of the model here. We built this model in the form of an LLM precisely to address this problem - to be able to utilize the textual data that accompanies the image such as the order history or clinical background e.g. patient demographics. Images and text are both embedded into the conversation, meaning the LLM can in theory respond using both.

Of course, there are lots of remaining challenges around integration and actually getting access to these data sources e.g. the EMR systems, when trying to use this in practice.

[crabbone]

My experience with working with hospital textual data is that, for the most part, it's either useless, or doesn't exist. The radiologist reading the image is expected to phone the specialist who requested the images to be red in order to figure out what to do with the image.

Hospital systems are atrocious for providing useful information anyways. They are often full of unnecessary / unimportant fields that the requesting side either doesn't know how to fill, or will fill with general nonsense just to get the request through the system.

It gets worse when it's DICOMs: the format itself is a mess. You never know where to look for the useful information. The information is often created accidentally, by some automated process that is completely broken, but doesn't create any visible artifacts for whoever handles the DICOM. Eg. the time information in the machine taking the image might be completely wrong, but it doesn't appear anywhere on the image, but then, say, the research needs to tell the patient's age... and is off by few decades.

Any attempt I've seen so far to run a study in a hospital would result in about 50% of collected information being discarded as completely worthless due to how it was acquired.

Radiologists have general knowledge about the system in which they operate. They can identify cases when information is bogus, while plausible. But this is often so much tied to the context of their work, there's no hope for there to be a practical automated solution for this any time soon. (And I'm talking about hospitals in well-to-do EU countries).

NB. It might sound like I'm trying to undermine your work, but what I'm actually trying to say is that the environment in which you want to automate things isn't ready to be automated. It's very similar to the self-driving cars: if we built road infrastructure differently, the task of automating driving could've been a lot easier, but because it's so random and so dependent on local context, it's just too hard to make useful automation.

[jarrelscy]

Thanks for the comments. I’m well aware as I’m also a practicing radiologist! Some hospitals in Australia where I work do a good job of enforcing that radiology orders are sent with the appropriate metadata but I agree that is not the case around the world. Integration, as always, remains the hardest step.

PS genuinely appreciate the engagement and don’t see it as undermining.

[robertlagrant]

I think this is too pessimistic. You can slowly add useful information that makes things more useful, if there's value in incorporating the information. I'm very familiar with EHRs and I get the problem, but it's not insoluble. And the full problem doesn't need to be solved to make progress.