[jerf]

"Give it examples that we consider moral"

That's a tall order, honestly. There's a lot of things in the current dominant SV philosophy that are fine and dandy and everybody thinks they agree with everybody else about them as long as everyone carefully agrees to not sit down and actually put numbers on the terms in question ("discrimination is bad!" "I agree!"), but when it comes time to write down concrete rules and provide concrete examples ("hiring a woman is 43.2% preferable to hiring a man; hiring an African American is 23.1% preferable to hiring a Chinese person") are going to make people squirm, and everyone involved in such a project is going to do everything in their power to avoid having to deal with the result.

I bet there's a number of people reading this post right now squirming and deeply, deeply tempted to hit that reply button and start haranguing me about those numbers and how dare I even think such things, as you've been trained to find someone to blame for any occurrence of such words and I'm the only apparent candidate. But I have no attachment to the numbers themselves and I pre-emptively acquiesce to any corrections you'd care to make to them, for the sake of argument. I expect a real model would use more complicated functions of more parameters, I just used simple percentages because they fit into text easily. But any algorithm must produce some sort of result that looks like that, and once you get ten people at a table looking at any given concrete instantiation of this "morality", 9.8 of them are not going to agree it's moral.

I cite the handful of articles we've even seen in peer-reviewed science journals, sometimes linked here on HN, which discuss the discriminatory aspects of this or that current ML system, while scrupulously avoiding answering the question of what exactly a "non-discriminatory" system actually is. It's one of those things that once you see it you can't unsee it. (And given that these papers are nominally mathematical papers by nominally "real scientists", if I were a reviewer I'd "no publish" these papers until they fix that oversight, because it isn't actually that useful to point out that an existing mathematical system fails to conform to a currently-not-existing mathematical standard.)