Can't agree with this. Even if it was just feeding a list to human investigators, who fairly investigated each case, there's still the significant issue of how the list is determined.
For example, the tax authority in your country right now catches a small portion of overall fraud. I give them a list of "hot leads" of fraudsters that's really just a list of all the ethnic minorities I hate. Obviously, even if they fairly investigate everyone, and nobody who's not defrauding loses out, this practice would be deeply racist and disproportionately punish the minority.
In short, we can't human-wash this problem at the end. Even if everyone punished was a fraudster, strong bias in the initial list is still clearly an issue.
Whilst you are right in general that biased algorithms are bad even when people using them are good, the problem in _this_ scandal was the the people using the algorithm were trigger-happy, and partially so in a biased way.
I only have as much information as is included in the original article, but this seems like a big issue to me:
>Having dual nationality was marked as a big risk indicator, as was a low income.
Again, even if these cases were all impartially investigated, you're still explicitly targeting low-income people and dual-nationals, probably significantly more likely to be an ethnic minority. I think the algorithm is a worthwhile story, not just the trigger-happy authorities.
That is indeed a big problem. I wouldn't be surprised if this marking as a risk factor wasn't even backed by data. (Note that even if it was backed by data, it would still be wrong).
> Note that even if it was backed by data, it would still be wrong.
Is it? This is the big question, as far as I can see. If my country has green people and blue people, and green people are overwhelmingly more likely to commit fraud, is it wrong to require additional fraud checks on the basis of being green?
Note that this isn't a conviction. It's accepted that the standard is not so exacting that no one innocent must be subject to investigation - that would be impossible under realistic circumstances. (Hell, even convictions don't meet that standard.)
On the other hand, I do accept the argument that you would have to be very, very cautious that you don't end up with a feedback loop if you did this - in other words, a system where you keep convicting green people because you investigate them, and the data therefore keep suggesting that green people have a greater propensity for crime, and so forth. That's undoubtedly happening in many countries, I'm sure.
I'll stipulate that in this scenario the data themselves are not actually biased. Whilst in reality these data often are biased by things like disproportionate policing and heavy punishhment.
Even then, discriminating by government based on race is bad even when a statistical basis for such discrimination exists. What makes "race" problematic to discriminate on is how easy it is to see "race". Or rather, how easily most people classify and distinguish between ethnicities based on how they look.
For an example, lets start with a small difference between blue and green people.
blue people are twice as likely to commit crime as green people, with a criminality rate of 0.2% vs 0.1%.
If this starts being how you police, if this 2x difference starts guiding decisions, then a lot of innocent people start being disadvantaged. The extra problem is that it is very easy to see if someone is blue or green. So it becomes really easy for a lot of people to start acting based on this 2x difference. This harms all blue people which is disproportionate. It then becomes a lot easier to get to the feedback loop you talked about.
Usually there is more nuance than that. It sounds high and mighty to take the principled position, but reality is that major offenders of this type of fraud are often ethnic gangs, and ignoring that signal is dumb.
Many companies do things like black hole traffic from Russia, Nigeria, China, etc, or make interactions higher friction (require MFA always, etc). Those tactics work and minimally impact real customers.
OK but are poor people and minorities more likely to commit benefits fraud? Because if so, I think that's fair game as an input, (obviously not if it's the only input).
Not sure about that in the real world. The temptation of "algorithms" is in reducing human workload which, realistically, means saving on salary costs by having fewer humans in the loop. The attraction of cost savings – and the externalization of repercussions – biases decisionmakers into believing that algorithms are a great solution. Yes, the real problem is abuse of algorithmic decisionmaking, and a lack of oversight, but in the real world you can't separate a technology from the people who use it.
For example, the tax authority in your country right now catches a small portion of overall fraud. I give them a list of "hot leads" of fraudsters that's really just a list of all the ethnic minorities I hate. Obviously, even if they fairly investigate everyone, and nobody who's not defrauding loses out, this practice would be deeply racist and disproportionately punish the minority.
In short, we can't human-wash this problem at the end. Even if everyone punished was a fraudster, strong bias in the initial list is still clearly an issue.