[throwaway97345]

An AI model is another word for a statistical model of reality, a small scientific theory, which actually works.

It is the most unbiased thing you can have. It's pure applied science/maths. You are only interested in the input that tell you something, and figure out which one to which degree.

You don't trust intuition or common knowledge. You don't trust anything that might introduce bias. You just look at the data and build the best decision mechanism that can be derived from it.

The degree to which you feel you need to "adjust" that is the degree to which you denounce science. You prefer your bias over science.

Today it's probably the cultural marxism / structural oppression narrative that inspires such manipulation. At other times it would have been infallibility of the papacy, or some other brain construct.

If you feel the need to adjust an ML model, be aware how strongly you feel that way. That's the degree to which you reject science.

Then proceed. Or reverse course. But be honest with yourself about what you're doing.

[IanCal]

You are missing the point of the entire article here. Quite aside from the fact that you can obviously have biased models because of their design or because of their training data, the key misunderstanding in your comment is this:

> You just look at the data and build the best decision mechanism that can be derived from it.

How do you define "best"? That's the issue. Not all errors are equal, not all distributions of errors are equivalent even if the total is the same.

[throwaway97345]

> How do you define "best"? That's the issue.

That's very simple with the example in the article. Who can pay back the loan best? You just fear the answer and rather twist up your reasoning.

[IanCal]

And that classifier is bad at determining who can pay the loan back when looking at the blue group.

Even using a very short term, entirely selfish view this can be bad for the loan company. It becomes clear that blue group people are being denied loans they could well afford, and so people in that group start moving over to other providers.

If the populations of blue groups are geographically clustered, this may mean losing large portions of business in certain areas, resulting in shutting down local offices if there's a physical presence (e.g. banks).

This is also entirely aside from legal concerns.

The best model is rarely found by simply optimising a basic measure with no context.

[XuMiao]

In terms of the statistical learning framework, the Bayesian optimal classifiers are NP hard to find under distribution free conditions. The models have rather large confidence intervals to compare with each other. Scientific methods can only lead you to a certain level of certainty, the rest are purely subjective.