[smoldesu]

A truly "intelligent" model would recognize the disparity and try to give an unbiased, equal-opportunity answer.

Unfortunately, these models are not really "intelligent". Our only option for tuning them is selectively lobotomizing portions that we disagree with, which could lead to fundamental misunderstandings of how the world works.

Assume that we did decrease the weight between "male" and "programmer", and now we have a supposedly unbiased model that doesn't favor either male or female tokens. Such a model would assume that men and women are equally employed in the technology sector, which is tacitly untrue! So, how can a machine actually understand reality then?

The simple answer is that it doesn't. None of this information actually helps it grok the real world. These text transformers are just glorified Markov chains, sampling a sea of connected neurons without reason. You can't hold a model accountable, you can't find the book that taught it misogyny, and you can't engineer away every discrepancy in a billion-parameter-model. Responsible uses of AI don't treat it like a human intelligence.