[KaiserPro]

> skeletal racial differences

£10 says that its not that. Anatomy is extraordinarily hard, and AI isn't that good, yet. Sure different races have different layouts, but often that's only really obvious post mortem. (ie when you can yank out the bones and look at them, there are of course corner cases where high res CAT/MRI scans can pull out decent skeletal imagery in 3D) There are other cases, but that should be easy to account for.

If I had to bet, and I knew where the data was coming from, I'd say its probably picking up on the style of imaging, rather than anything anatomical. Not all x-rays have bones in, and not all bones differ reliably to detect race.

> keep politics out of science.

Yes, precisely, which is why the experiment needs to be reproduced, and theories tested through experimentation. The reason why this is important is because unless we workout where this trait is coming from, we cannot be sure the diagnosis is correct. For example those with sickle cells have a higher risk of bone damage[1] which could indicate they are x-rayed more. This could warp the dataset, causing false positives for sickle cell style bone damage.

[1]https://www.hopkinsmedicine.org/health/conditions-and-diseas...

[tedivm]

>If I had to bet, and I knew where the data was coming from, I'd say its probably picking up on the style of imaging, rather than anything anatomical. Not all x-rays have bones in, and not all bones differ reliably to detect race.

This was my guess as well. I've spent a lot of time around radiology and AI (I used to work at a company specializing in it) and we read a lot of the failure cases as well. There was one example where the model picked up on the hospital, and one hospital was for higher risk patients- so it learned to assign all patients from that hospital to the disease category simply because they were at that hospital.

There are a ton of cases like this out there, especially when using public datasets (which in the medical field tend to be very unbalanced datasets due to the difficulties of building a HIPAA compliant public dataset).

[krona]

> one hospital was for higher risk patients- so it learned to assign all patients from that hospital to the disease category simply because they were at that hospital.

That just sounds like poor feature selection/engineering. Garbage in, garbage out.

[tedivm]

Yeah there are definitely ways they would have avoided that, but it's just one example of many. The whole point of ML is that it picks up on learned patterns. The problem is that it can be difficult to identify what it is learning from- this paper itself says they do not know what is causing it to make these predictions. As a result it is difficult to validate that the model is doing what people think it is.

[MontyCarloHall]

>I'd say its probably picking up on the style of imaging, rather than anything anatomical

Certainly possible! They do control for hospital and machine …

>Race prediction performance was also robust across models trained on single equipment and single hospital location on the chest x-ray and mammogram datasets

… but it’s also possible that different chest x-rays were being used for different diagnostic purposes and thus have a different imaging style, which a) may correlate with ethnicity and b) does not appear to be explicitly controlled for.

[ASalazarMX]

The weak factor in these AI studies seems to be data set normalization.

[knicholes]

I wonder if some communities use certain x-ray machines verses which machines are commonly used by other communities and this has nothing to do with race but the machine being used. I read over the paper but didn't really understand it. Maybe all this is doing is identifying which machine was used.