[sucessfullylate]

How is being able to determine what racial/ethnic group a person belongs to from raw data regarding their medical history/information wrong in any way?

This is not about equal rights, voting access, or systemic racism. Rather, the health trends that underlay specific groups of people.

[retrac]

Our cultural definitions of race/ethnicity often have little to do with someone's /actual/ genetic ancestry. For example, in the USA, Australian indigenous people, Andamanese islanders, predominantly European mixed Euro-African Americans and many others would all just get, I suspect medically unhelpfully, labelled as black.

If the AI is using this information (probably informed by some biases it's picking up from the training data) then unless those biases are actually correct and justified we're training a system that's going to be using racial data, potentially inappropriately (see examples above and in the article), in rendering medical advice or diagnoses. Now that's a problem.

[raxxorrax]

But your solution is the cause of the problem in the first place. An AI doesn't attribute anything that isn't in the data. Feedback from medical indications is certainly required, the racism would solve itself through that.