[noir_lord]

> Why would we want to reproduce existing structures of oppression in mechanical form?

If (for example) 66% of Doctors are male and 34% female then it's not reproducing "existing structures of oppression" it's inferring something about reality.

[notahacker]

In an environment in which Blue people are banned from becoming doctors, its also inferring something about reality to conclude that 0% of Doctors are Blue. It would be entirely wrong, however, to use these inputs to infer anything whatsoever about the respective propensity of Blue and Green people to become doctors in an environment in which such a rule or idea of a rule had never existed. Obviously "structures of oppression" - real and imagined - which lead to fewer female doctors even in western liberal democracies where women wishing to become doctors are generally met with encouragement are less extreme, but that isn't to say they don't exist or that a computer output (or human interpretation of said computer output) is likely to draw correct inferences from it.

And if you think that people won't use the idea that the outputs are unbiased because the computer isn't programmed with the same prejudices that produce the inputs, I have some algorithmically-generated investment advice involving a bridge to sell you

[pekk]

> It would be entirely wrong, however, to use these inputs to infer anything whatsoever about the respective propensity of Blue and Green people to become doctors in an environment in which such a rule or idea of a rule had never existed.

That's fine but it isn't the goal of these algorithms. It isn't the reality that is useful for them to learn. It's a different problem to try to build some kind of "unbiased" ontology rather than just to learn about words. Feel free to research or create solutions to this other problem, it sounds interesting.

[arrrg]

If it’s saying that and only that then that’s obviously fine. However, if that knowledge is then applied in any other way, then that’s problematic.

[anigbrowl]

It's inferring something about reality, but what?

Suppose, for example, that I gave this same statistic to someone and then asked them to select from a pool of 100 applicants for 50 available places in medical school. Let's assume that there's an equal # of male and female applicants and that their exam results are all similar. Do you think that knowing about this 66-34 split might influence the gender balance of the final selection?

[newsbinator]

Knowing about the gender balance wouldn't influence the final selection if you programmed the selection criteria not to be influenced by the gender balance.

The whole point of training and using machines is to make more accurate, more useful decisions in a complex world.

That can't happen if we give them data that isn't borne out by reality, or tell them to ignore data that is.

[anigbrowl]

If you have to change the terms of the question to give an answer, then I think I've made my point.