[aetherson]

I think Dan performs a mild sleight-of-hand trick here: he asks why we don't consider this a bug when any other software would consider it a bug. But in fairness, the question was not, "Did this prompt have a bugged output," it's "Did it have a racially biased output," and that's a more emotionally charged question.

If I wrote software that choked on non-ASCII inputs for say a name, and then someone said, "Hey, that's a bug," cool, yes, fair. If someone said, "This is evidence of racial bias," I mean... I'd probably object. Even if there is some nugget of truth to the idea that there may be some implicit language bias which relates to race in there.

I think Dan does a decent job showing that there is some level of racial bias -- not on the level of "the model refuses to show asian people," but on a level of "the model conflates certain jobs with certain races," and that's fair. But I just found the lede of, "Why don't people admit there's a bug in AI" to be a little obtuse.

[mlyle]

> that there may be some implicit language bias which relates to race in there.

I think there is some stuff in the middle. I think disadvantaged groups deal with more of these bugs by being underrepresented in the teams that design this stuff.

Are soap dispensers that don’t give soap to people with darker skin racially biased? Kind of.

Especially once we keep getting adverse outcomes and don’t manage to prioritize fixing it.

[kelseyfrog]

> Are soap dispensers that don’t give soap to people with darker skin racially biased? Kind of.

I'm struggling to extract the nuance that I feel is embedded here. Is it an ontological issue, a quantitative, or a qualitative dimension that puts it in the realm of kind-of?

[malfist]

Vision based tech developed in silicon valley notoriously doesn't often include a diverse training sample. There are have been more than a few cases were technology has only been tested on light skinned people and released to the world and didn't work for anyone with more pigment in their skin.

Soap dispensers that detect hands are one example, highlighted by GP, I believe Microsoft's Kinect also had issues on release detecting non-white people.

[Cheer2171]

Yeah, why "kind of" and not "absolutely"?

The black box systematically changes the output based on the presence of an input signal of skin tone, which is strongly correlated with race.

Same with photographic negative film, where engineers made a trade-off with the chemicals that made contrast much better for lighter-skinned faces, which made darker-skinned faces much worse. In group portraits, darker-skinned people would appear as just teeth and eyes. Although in true American style, it was chocolate and furniture companies (and their advertisers) who pushed Kodak to develop better technology for capturing dark brown colors.

https://www.nytimes.com/2019/04/25/lens/sarah-lewis-racial-b...

[mlyle]

I think it's "kind of" because part of the problem is intrinsic; if you're trying to detect reflections, and you're not even reliable with light skin 100% of the time, it says that there's some inherent difficulty here.

You realize it's biased when you think about it and realize the product could never have successfully shipped if it offered white people the poor reliability that it has for darker skin.

[tacitusarc]

There is a large amount of academic literature claiming racial groups are purely ethnocultural, and in that light it is ridiculous to claim a soap dispenser is racially biased.

[wizzwizz4]

The US has its own special meaning of "race", "racial group" and similar – but, since societal groups are what society thinks they are, this means that "racial groups" in the US are (in large part) defined by things like skin colouration. In that context, claiming that the soap dispenser is "racially biased" is a perfectly good description.

If you perform a root-cause analysis (a soap dispenser doesn't exist a vacuum), you will discover "racial bias" in the structures and institutions that designed and created the soap dispenser. Metonymically, therefore, it's even more accurate to claim that the soap dispenser is "racially biased".

(Author's note: I continue to reject the US's conception of "race". Please do not understand this comment as legitimising it in any way.)

[mlyle]

Say you're making a soap dispenser. It doesn't work for some people. You don't really care or notice because they're members of racial groups that you actively dislike or maybe just don't really care about.

Couldn't the soap dispenser still be an instrument of racial bias, even if this racial group is an ethnocultural distinction?

[tivert]

> I think Dan performs a mild sleight-of-hand trick here: he asks why we don't consider this a bug when any other software would consider it a bug. But in fairness, the question was not, "Did this prompt have a bugged output," it's "Did it have a racially biased output," and that's a more emotionally charged question.

In the initial examples, it's not a slight of hand. The thing has racial biases so bad that they bugged the output. It's job was to convert a casual photo of a particular person into "professional" photo, and instead of just changing the clothes and setting, it changed the person too.

Then all kinds of apologists tried to gaslight the bug away instead of acknowledging the system is faulty and not fit for purpose.

> If I wrote software that choked on non-ASCII inputs for say a name, and then someone said, "Hey, that's a bug," cool, yes, fair. If someone said, "This is evidence of racial bias," I mean... I'd probably object.

And what if they just said it was an instance of "bias," like the OP describes similar bugs? I don't think you'd have grounds to object.

[bo1024]

I think the article does a nice job connecting these bugs to other historical, non-AI bugs that result in a racially biased system.

[nateglims]

But the LLM didn't do what it was asked

[michaelt]

In the example of "upload a photo of an asian person, ask the model to convert it into a professional linkedin profile photo, race gets changed" that's the model not doing what it was asked, yes.

In the example of "asked for images of a chemistry professor many times, always a white guy wearing glasses" on the other hand the model does what it's asked, in that the images do look like chemistry professors. However the output demonstrates gender and racial bias.