[throwaway1851]

It’s a really thorny set of issues. I remember reading a discussion recently about gender bias in coreference resolution. (Coreference resolution is the task of linking words such as “him” or “her” or “the company” to other words in a text, such as “the barista”, or “Jane”, or “Microsoft”.)

The findings were that the language model did a worse job of performing coreference resolution in gender-reversed situations (eg, male nurse, female firefighter). There was an example of a sentence like: “The nurse told the patient he would be leaving soon,” and the model was more likely to link “he” to “patient” because of the biased perception that a “he” is not likely to be a nurse.

What stuck with me was a claim that the model was using bias rather than “the evidence of the sentence” to perform the task. This seems purposefully ignorant of how language works: the perceived probability distribution of genders over occupations (even if biased!) is a part of the global context that imparts meaning to language. Fiddling with the data to get the model to become unaware of such context arguably changes the tool from being a model of language to a model of some ideal of what language could be.

To be clear, I’m not criticizing efforts to detect or mitigate bias in training data. Oversampling gender-reversed texts could indeed make a much better performing model, and a fairer one. I just think there’s a real issue with imparting top-down value judgments into these processes and pretending that they aren’t value judgments.