[GardenLetter27]

> None of these features explicitly referred to an applicant’s gender or racial background, as well as other demographic characteristics protected by anti-discrimination law. But the model designers were aware that features could be correlated with demographic groups in a way that would make them proxies.

What's the problem with this? It isn't racism, it's literally just Bayes' Law.

[crote]

Let's say you are making a model to judge job applicants. You are aware that the training data is biased in favor of men, so you remove all explicit mentions of gender from their CVs and cover letters.

Upon evaluation, your model seems to accept everyone who mentions a "fraternity" and reject anyone who mentions a "sorority". Swapping out the words turns a strong reject into a strong accept, and vice versa.

But you removed any explicit mention of gender, so surely your model couldn't possibly be showing an anti-women bias, right?

[alternatex]

I've never had any implication of my gender other than my name in any CV over the past decade.

Who are these people who make a career history doc include gender-implicating data? And if there are such CVs, they should be stripped of such data before processing.

The fraternity example is such a specific 1 in a 1000 case.

[crote]

Just because you aren't aware of it, doesn't mean it isn't there. There are plenty of less on-the-nose examples a model can accidentally train itself on.

Into horseriding? Probably a woman. Into motorcycles? Probably a man. Into musical theater? Probably a woman. Into football? Probably a man. Worked part-time for a few years in your 30s? Probably a woman. I could go on and on for hours, as there are relatively few hobbies and interests which have a truly gender-neutral audience.

The problem isn't obvious bias. Everyone can see those and filtering them out is trivial. It's the subtle proxy values which are risky, as you have to be very careful to avoid accidentally training on those.

CVs should ideally indeed be stripped of such data, but how do you propose we verify that we stripped it of all potential proxies? And what's going to be left to train on after stripping?

[alternatex]

I feel like what will be left is only the things that are good indicators of professional competence. Of all things that are fluff in a CV, a hobbies section would probably be #1. Not to mention probably a red flag for any CV reviewer.

[triceratops]

> I've never had any implication of my gender other than my name in any CV

So you're not implying gender other than by implying gender? If humans can use names to classify people into genders, a model can do the same thing.

[alternatex]

It's information that's easy to strip before running it through machine learning.

The implication in the parent comment is that CVs are inherently bound to gender and I cannot see that to be the case for most.

[Viliam1234]

> It isn't racism, it's literally just Bayes' Law.

That may be logically correct, but the law is above logic. Sometimes applying Bayes' Law is legally considered racism.

https://en.wikipedia.org/wiki/Disparate_impact

[anonym29]

Legal frameworks can indeed contradict mathematical optimization functions, statistical patterns exist independent of our social preferences about them, and aggregate behavioral differences between groups (whatever their causes) will produce disparate algorithmic outcomes when accurately measured.

If certain demographic groups legitimately have higher base rates of welfare errors (due to language barriers, unfamiliarity with bureaucratic systems, economic desperation, or other factors), then an accurate algorithm will necessarily produce disparate outcomes.

If we dig deeper, there are three different underlying questions that are attempting to be addressed by the authors of this "fair" fraud detection system -

1. Do group differences in fraud rates actually exist?

2. What mechanisms drive these differences?

3. Should algorithms optimize for accuracy or equality of outcomes?

The article conflates these, treating disparate outcomes as presumptive evidence of algorithmic bias rather than potentially accurate detection of real differences.

Pattern recognition that produces disparate outcomes isn't necessarily inherently "broken", it may be simply be accurately detecting real underlying patterns whose causes are uncomfortable to acknowledge or difficult to address through algorithmic modifications alone.