| I am not sure the data model is the problem here.
I feel like the journalist tried really hard to make a case against scoring (which I do not like either), but overlooked the fact that the whole system in which its embedded is bad.
The case should not be on the technology but on its usage. It’s already looking like a bad journalism piece in the first part : ” Being flagged for investigation can ruin someone’s life, and the opacity of the system makes it nearly impossible to challenge being selected for an investigation, let alone stop one that’s already underway. One mother put under investigation in Rotterdam faced a raid from fraud controllers who rifled through her laundry, counted toothbrushes, and asked intimate questions about her life in front of her children.” Here the problem is not the algorithm, its the investigators. Another ethical problem for me : the system of flagging in whole relied partly on anonymous tips from neighbors. I am not an expert but I feel more at ease about a system that rely on a selection algorithm + randomness than delation. I think the problem was the processes around the algorithm not its existence itself.
The journalist seems to assume during the whole piece that the algorithm will become the main/only way to identify fraudsters. If its the case, it’s terribly wrong because how are you training your algorithm then ? Most of the time, the piece try to put the reader in an emotional state of fear and anger and is not at all doing any analysis, while faking it using a lot of numbers and graphs. Sorry for the long rant but I am surprised that this came from Wired which I consider quite good on tech topics, and that its on HN 2nd page. I am against government scoring and algorithms for legal / police cases precisely because it can be badly used by powerful people. Am I the only one to feel that its not a good article ? |
Indeed. Going deeper, the 'problem' is a social/cultural belief that doing X at scale, using a computer, is somehow more ethical than having a bunch of people do the same immoral thing. Computer automation and algorithms become a moral justification (or at least a Hail Mary) for immoral acts.
There is at once a diffusion of responsibility, a causal disconnection of consequential acts, a reassignment of responsibility, and - given that we bow down to machines as our masters rather than our tools - a change from choice to a belief in the "inevitability" of unstoppable processes.
Together these make us no longer question whether:
1) Computers are more reliable, consistent and accurate than people
2) Computers are fairer/just
3) Computers are more comprehensive/inclusive or selective/prejudicial
4) Computers are actually economically more effective
Of course, this has been going on since the 1960s at least and was part of "systems analysis for automation". I think we have regressed. Whereas it was commonplace to sceptically question technology in the 60s and 70s, today we start with the assumption that it must be "better" and then have to figure out how maybe it's not.