[dredmorbius]

With machine learning, the model isn't even a set of explicitly selected factor whose parameters are then derived from data, but hidden factors (and interactions) within the neural net. The engineers who construct the models don't even knnoe how they achieve their ressults, only that they do.

Upside? Plausible deniability! An NN AI that happens to score based on age, gender, gender preference, religion, disability status, and ethnicity, but buried among other factors and not explicitly exposed? Perfectly fine! (At least until you can hire stats PhDs to prove statistically significant bias.)

Bad news? The engineers have no idea why (or when) the model gives bad results. It's answers without insight.