[heavyset_go]

This is why algorithmic and predictive policing is so terrifying. Everyone in the justice system now has a great way to shirk liability for their actions and decisions: they can just defer to the infallible black box system that tells them who is a criminal or not. Want to side step accusations of bias or targeting? They were just going after whoever the black box said was guilty, and computers purportedly are not biased.

The corollary to this shirking of responsibility is how it lets people, and not just law enforcement, justify abuse of whoever the system says is guilty of a crime, because after all, the company that made the black box says that there is a one in a trillion chance of encountering a false positive. Judges will throw the book at defendants because, statistically, the black box system is almost never wrong, and the system says the defendants are monsters.

But the most Kafkaesque part is that people will never get an answer for how these systems determined they were guilty. It's a trade secret, it's part of on-going investigations, it's critical to national security, or in Apple's case, it's literally illegal to look and find out how their system came to conclusion because viewing the data itself is a crime. These systems often inscrutable ML models, as well, and we all know just how buggy and error prone computers and software can be.