[throwaway07Ju19]

> There’s nothing in the documentation to indicate what criteria the the HPPD would use, or is currently using, to add a face, license plate, or smartphones to a “blacklist,”

Regulation can't come soon enough for this industry.

About 20 years ago one of the youngest cops ever to be promoted to detective in the City of Chicago told me this after a long night of drinking...

"If I see a car with more than two black people, I just followed it. Sooner or later they'll make a mistake and I can pull them over. Very often one of them will have an outstanding warrant.". When my face turned to horror he asked me the jaw-dropping question "you are Italian, right ?"

Honestly, if ML is statistically driven, how long before it determines that African Americans and Latinos are over represented in the criminal database and should have a higher blacklist score as a rule and the whole cycle repeats. Meanwhile innocent looking white kids can keep selling meth in the park.

[discardable_dan]

This is sinister in a subtle way: profiling in this capacity is not only wrong, it is self-perpetuating. If police are trained to do this, then more African Americans are arrested even without warrants, and then end up in the system / in legal trouble, which means more are likely to have warrants. In turn, this means that future police trained to do this are likely to find success doing it by virtue of past police doing it.

That's... totally fucked.

[throwaway2048]

Thing is, you don't even need overly racist whoever for this to happen, a completely "neutral" neural net would happy take the same stroll down that particular primrose path without tightly controlling for self-reinforcing effects like this.

[justinjlynn]

Evolutionary systems are very good at perverting any and all definable fitness functions - it's a red queen's race. At best, if you want systems to behave you either have to supervise them all the time or have a system which can which itself must be supervised less often. While we want human morals and ethics to be a part of our systems, humans will forever need to remain in the loop.

[marvin]

It’s fascinating to see that we’re _already_ at a point where the AI alignment problem («what morality do we want our automated systems to have») is becoming a practical reality, long before AGI or self-improving AI appears to be imminent.

I think this is a clear, contempirary demonstration that philosophizing around the use of AI (even today) has practical benefits beyond the practice of philosophy as an inherently worthwhile species-wide intellectual pursuit.

[jacobolus]

> AI alignment problem

This is a problem as old as civilization.

Bureaucratic procedure (of governments, religious organizations, corporations, armies, NGOs, ...) always leads to bad outcomes in edge cases if individual administrators are not given enough autonomy to make exceptions and enough oversight/accountability to prevent abuse.

[marvin]

I think it’s also fascinating how this is an age-old philosophical problem, as in the stories of the malicious genie. People have known for a long time that the rigid, precise specification of morality or desirability isn’t easy.

[Spearchucker]

Is it so hard to remove gender race nationality and so on from automated profiling?

[throwaway2048]

There are tons of stats that serve as effective proxies for such things: What schools did you go to, what is your name, who are your friends, etc.

So yes, it is EXTREMELY hard.

https://www.reuters.com/article/us-amazon-com-jobs-automatio...

Thats exactly what amazon tried to do here, but because the AI based its decisions on amazon's previous hires (that is, almost no women) and blinded the AI to gender, it instead proxied women for one of a dozen other things such as school, courses taken, etc.

[Ididntdothis]

That’s the problem with ML and statistics. With the wrong input they quickly turn into something that constantly reinforces the status quo. It’s a dangerous path if not handled carefully.

You can see this even in advertising. All the ads I am being shown are about stuff I have bought or was interested in. They can’t give me new ideas or handle when my tastes have changed.

[throwaway2048]

Yep, a related question you might ask is: Would machine learning end slavery, or perpetuate and strengthen it, assuming it existed today as an accepted mass societal institution.

Consider it would likely be wealthy slaveholders that would aggressively deploy it first, being as they hold most of the capital, and have the most to gain from it.

I think people should consider the answer to that question very seriously.

[bko]

From your story it seems like human guards are profiling African Americans and Latinos now. At least with a bot you'd be able to explicitly prevent them from doing so. Probably a lot harder to prevent a human from using biases

[closeparen]

Machine learning - using statistical associations between features in a large population to make predictions about individuals - is profiling. There is no such thing as removing profiling from machine learning. The profiling is the only thing there.

[ceejayoz]

> At least with a bot you'd be able to explicitly prevent them from doing so.

Not really. They're really good at finding a proxy for the banned characteristic.