[SpicyLemonZest]

It seems like the reason the researchers in this paper are concerned is precisely that they tried and failed to understand how the ML algorithms are doing this. If they’d discovered that white people have a subtly distinctive vertebra shape the model was detecting, it would have been much more of “oh, we discovered a neat fact”.

[civilized]

I don't think they tried very hard at all. I see no meaningful use of modern explanation tools.

There are lots of known ways in which people of different races are different physiologically. Probably even more unknown ways.

There could also be differences in imaging technology used in different communities, as others have suggested. I'd be a bit surprised if something like that could create such a strong signal but it's on the table.

[SpicyLemonZest]

For those of us less familiar with this space, what are these modern explanation tools? (I certainly agree that it's plausible the model is seeing a physiological difference, and the researchers seem to have considered a few concrete hypotheses on that dimension.)

[civilized]

Here's an introduction to one technique: https://cloud.google.com/blog/products/ai-machine-learning/e...

This is a cutting edge subfield of ML, so it's understandable that one paper in a medical journal isn't going to be on that cutting edge, but I think they should at least acknowledge that their investigations barely scratched the surface.

[SpicyLemonZest]

Thanks! This looks pretty neat, I'll have to dig more into their explainable AI product later.