[joshuamorton]

>Framing this problem as "bias"

Except that's exactly what it is. Much as your model was biased against interns.

> and especially hyper-focusing everyone's attention on diversity aspect of it is extremely irresponsible.

Why? Pointing out a specific and concrete harm badly designed ML models cause is irresponsible? Just because the same kind of methodological flaw can cause other harms its irresponsible to use a motivating example?

[gambler]

>Why? Pointing out a specific and concrete harm badly designed ML models cause is irresponsible?

In my opinion, yes, if it leads most readers to misjudge some fundamental properties of the problem as a whole. Again, I'm not saying this article is guilty, but most are.

[joshuamorton]

> In my opinion, yes, if it leads most readers to misjudge some fundamental properties of the problem as a whole.

Which problem? The general statement of this problem is "models, trained on [somehow] misrepresentative data [or even technically representative data] can draw unintended conclusions that lead to harm". Specifically in this case, the harm was "the model was basically just trained to ignore all women applicants due to bad inference of conditional probabilities".

This is a common thing. Because our society draws lines and has bias, its fairly common for modelling failures to exist along those lines. Indeed, sometimes the failures are mostly harmless and immediately obvious, but often they aren't. And people building models should be made aware of those failure scenarios, and be especially aware of failure scenarios that affect underrepresented groups, because those groups are the most likely for the model to fail on if you aren't actively looking for them.

And this stuff is pervasive. Facial recognition tech is much worse at noticing the faces of darker skinned people [1]. Some of this is because the people building the common models (eigenfaces etc.) didn't use diverse skin tones, but some of it goes back further, white balance in film was tuned for lighter skin tones until the 90s[2]. Some of that has likely persisted into modern film and camera technology, unfortunately. People working with data need to understand their data. And that means understanding how bias infests their data.

> fundamental properties of the problem as a whole

You've yet to state the "whole problem" or the fundamental properties that people might misjudge. So I'm unclear what they are.

[1]: Arguably an advantage now.

[2]: https://petapixel.com/2015/09/19/heres-a-look-at-how-color-f...

[gambler]

>Which problem? The general statement of this problem is "models, trained on [somehow] misrepresentative data [or even technically representative data] can draw unintended conclusions that lead to harm".

Throwing AI at answering an ill-formed question or optimizing a process that shouldn't happen in the first place is not something that can be corrected by getting better training data.

Moreover, automation can have consequences that aren't detectable by analyzing some test set.

[westoncb]

> Except that's exactly what it is.

Using the term 'bias' has certain political motivations behind it. It's not about the term being technically untrue as it is about the term being non-neutral. For instance, here are some definitions of 'bias' I just grabbed from American Heritage:

"A preference or an inclination, especially one that inhibits impartial judgment."

"An unfair act or policy stemming from prejudice."

"A statistical sampling or testing error caused by systematically favoring some outcomes over others."

The ML model does not have a preference, inclination, or prejudice relating to interns, except insofar as we anthropomorphize it to have them. What does using a word suggesting that add?

A more neutral account of what's going on is along the lines: It's easy to accidentally train ML models so that they will make systematic errors. (Among those errors is the possibility for it to exhibit behavior resembling prejudice.)

[joshuamorton]

Fine: it's easy to accidentally train ML models so that they will make systematic errors. Often these errors stem from systematic biases in our society, model creators should therefore be aware of the potential biases[1] that their models could reflect, and how to prevent them.

[1]: With the political motivation.

[westoncb]

> Often these errors stem from systematic biases in our society ...

Depending on the what the appropriate quantification of 'often' is, that might make sense. Do we have enough reason to believe it would take on a high enough value to merit the usage of a term that refers only to it?

The other problem with what you're describing is that all we actually know is that the model is reflecting the current state of things. Your statement attributes particular causes to the current state of things, and implies a certain valuation of the current state of things (which I don't personally disagree with, necessarily—but I don't think my personal views should be reflected in scientific/engineering jargon).

So given the uncertain value of 'often,' and the unsettled nature of the causes behind various aspects of the 'current state of things,' it seems to be solidly jumping the gun to frame the entire general problem with a term that refers to this partial and fraught aspect of it.

[joshuamorton]

>Your statement attributes particular causes to the current state of things

I didn't, nor should it matter how we got to where we are for a builder of a thing.

> and implies a certain valuation of the current state of things

This may have happened, but I'd disagree: recognizing that there exists inequality doesn't cast value judgement on that inequality. I simply stated that they're there. Perhaps saying "how to prevent them" is casting value judgement, so I might walk that back, model creators should be aware of the biases and aware of tools and strategies to account for them, if so desired.

Personally I think you're a bad person if, armed with the tools to detect and correct, you decide its okay to build something that has a systemic error that wrongly disfavors some group. But perhaps that's just me.

[westoncb]

> ... recognizing that there exists inequality doesn't cast value judgement on that inequality.

You just asserted your attribution of cause right there: inequality. There are multiple possible causes for differing demographic representations in various roles. This is not a settled issue, even though people on both sides promote competing ideologies to the effect that it is.

(And again, I have intentionally left my own views on the subject out of this, even though I suspect they align with yours (insofar as cause attribution goes): I'm just pointing out the fact that this isn't something society agrees on, nor is it something the scientific data resolves unambiguously.)

> Personally I think you're a bad person if, armed with the tools to detect and correct, you decide its okay to build something that has a systemic error that wrongly disfavors some group.

Agreed, hinging on that point about cause attribution.

[belorn]

> Often these errors stem from systematic biases in our society

No, this does also not match.

One of the easiest way to get a ML model that creates systematic errors is spam filters. If I take my spam folder with no consideration, what the filter will learn is that any language which isn't my own are spam, and that servers located outside my nation are spammers. This resembles prejudice.

The cause of this systematic error is that individual email addresses do not get ham emails uniformly from every nation and every language. Proximity warps the data. I would need to normalize the data based on language and nation if I wanted to remove those errors in the filter. Looking at it from a political perspective does not make the filter perform better, and fixing it from that side has a high risk of causing even more errors in the model.