[retrac]

Our cultural definitions of race/ethnicity often have little to do with someone's /actual/ genetic ancestry. For example, in the USA, Australian indigenous people, Andamanese islanders, predominantly European mixed Euro-African Americans and many others would all just get, I suspect medically unhelpfully, labelled as black.

If the AI is using this information (probably informed by some biases it's picking up from the training data) then unless those biases are actually correct and justified we're training a system that's going to be using racial data, potentially inappropriately (see examples above and in the article), in rendering medical advice or diagnoses. Now that's a problem.

[raxxorrax]

But your solution is the cause of the problem in the first place. An AI doesn't attribute anything that isn't in the data. Feedback from medical indications is certainly required, the racism would solve itself through that.