| There are some very real effects that need to be acknowledged in order to design a good system. Here's a hypothetical example I gave in a similar thread here a few months back. It's loosely based (Though not too far removed from reality!) on some real-world systems that failed to address this: Suppose you are designing a facial recognition system for police to use in the field while investigating a recent crime to see if anyone with a criminal history is nearby. (Data taken from: https://en.wikipedia.org/wiki/Incarceration_in_the_United_St...)
Because blacks are over-represented in the US criminal justice system (40% of the prison population vs 13% of the population) and because part of what defines "black" is the outward appearance of certain facial features, a facial-recognition algorithm which is trained to recognize criminals, with a cost function based on prediction accuracy alone, and facial features as input parameters is likely going to have false positives that over-represent blacks. It's very important to consider this when you develop a training set. The developer error (who mostly failed to understand Baye's theorem here) might work something like this: They take 100 innocent people's faces at random. (On average it will have only 13 blacks) Then take 100 random criminal faces from inmates. (On average it will have 40 blacks.) Then mix up the groups into your training set and assign a prediction score 1 or 0 depending on whether or not your classifier has correctly predicted whether or not a face was in the criminal group. Then, based on no other feature than race, your neural net can get better performance based solely on guessing more often that black people are criminals. That's not a good thing. In fact, if it's looking at a black face from its training set, the odds are nearly 2 to 1 that it's one of the criminals, even though the odds that are at least 2 to 1 against a random black person having a criminal history. The likelihood of being falsely identified as having a criminal history is much greater based on the only variable of being black. And this type of thing has happened several times already in production systems! Conversely, the same system, trained on the same data in the same training set, can get higher performance than random by simply guessing that any non-hispanic white person does NOT have a criminal history. Thus, it's pretty important to correct your training set to reflect the correct Bayesian prior, and the underlying structures that sometimes go by the label "structural racism" or "institutional racism" are essentially exactly that reality in this case. |
To be honest, I don't think our understanding of these systems is mature enough for us to be just throwing them out into our societal systems right now. There needs to be a lot more testing because the possibility of unethical results is pretty damn high.
That said, I still think that the concept of structural racism is a bad way to look at this problem, as it's simply one form of a common error when looking at sample sets.