[visarga]

A great post. I can attest the same about information extraction from semi-structured documents. The situation is far from full autonomy. Can't do anything without human in the loop, not even with the latest AI.

I am seeing this trend - everyone can explain at length why AI is "not quite up there" in their own field, but believes it's "near AGI" in other fields. We find it hard to imagine future difficulties AI will have to face in general, we can only do that in our own field where we have learned from direct experience.

[haldujai]

Exactly this! I chose radiology given my background thinking I could "easily build radiology AI systems" and help our struggling system.

Then I became a radiologist and quickly discovered how hard this is. Something "as simple" as NER and entity-linking on radiology reports is damn near impossible at the moment (even with SOTA LLMs which have made it easier but still not accurate enough for production use).

[chaxor]

NER, Entity linking, and relationship extraction definitely seem to be 'low hanging fruit' due to LLM improvements, but one of the big problems is that they really need a completely different architecture to limit the decoder vocab if using a decoder transformer for producing the set of sequences in relation extraction with specific entity ids. A Longformer with full attention to input sequence, and sliding window attention to a large dictionary could be a decent way to find tune a system like this, but there are few that try it. Unfortunately there's a lot of stupidity going around right now in thinking the answer is just to 'pRoOoMpT tHe LLm RiGhT', but that will always be exceedingly wasteful such that processing terabytes of files will be prohibitively expensive, and there's no guarantee the system will always restrict to the specific vocab and structure desired.

The images in radiology definitely make these types of things harder, and the sparsity is an enormous issue. However, working with some projects in this area, I don't think it's as impenetrable as a lot of radiologists in AI suggest. The main thing needed in the field is adoption of better techniques and architectures to deal with these problems.

[haldujai]

I agree it's not impenetrable, that's why I'm working on this problem. What I disagree with is the "this is trivial" statements.

> Unfortunately there's a lot of stupidity going around right now in thinking the answer is just to 'pRoOoMpT tHe LLm RiGhT'

I agree with that this is not the right approach despite all the media hype, my research has been (more or less) attempting what you've proposed.

> A Longformer with full attention to input sequence, and sliding window attention to a large dictionary could be a decent way to find tune a system like this, but there are few that try it.

Good idea, although I'm biased as we tried this ourselves! Problem is the dictionary (ontology) doesn't exist. RadLex and UMLS are far too inadequate in coverage. Actively working to address the gaps and hope to have something to open-source within the next couple of months.

[bick_nyers]

I previously worked for a Radiology PACS and it's hard to get funding/interest to even tackle the problem. With how lucrative it could be, I would think that a corporation would be very interested in putting resources into it, but this has not been my experience.

No PACS that I know of even wants to tackle digital pathology in a significant way, which last I heard had about 5% adoption versus glass slides.

[haldujai]

There are several corporations, some of which have raised > 100M, tackling this problem. AIdoc being one of the more notable ones.