| This is a serious situation that has been unfolding for years now in the world of fraud in general (from both the retailers and the banks' perspective). I used to work in a company doing fraud prevention. The first thing that shocked me when I started in the role, was the sheer indifference people working there had for rejected transactions. The way these were supposed to be handled was like this: 1. Transaction is rejected because of a rule/condition/bank decline
2. Investigate transaction manually and identify root cause
3. Contact customer and provide information on how they can resolve this (ask for more details if you are unsure, mark them as safe to allow transactions as it was a clear false positive or ask them to speak to their bank or payment provider as the system wasn't the one that rejected the transaction) The general view was though that if the customer think it's important, then they should get in touch with the company and there shouldn't be proactive work done on these transactions since chances are "most of them are dodgy". The problem was that there were thousands of these every single day and a very small team working in that department - less than 10 people. I voiced the fact that even if someone is fast at identifying these and going through them (once you developed experience in identifying the cause of the decline and doing the administrative work of allowing the user to complete transactions, emailing them and then keeping an eye on their account for when they try to make a purchase) they couldn't work through more than maybe 2-300 transactions in a day. That was 2-300 of someone working ONLY on this task, almost automatically and having a deep understanding of what happened in each of them. Realistically it was more around 50-100 transactions per day. The rejection list could have up to 3-4000 entries per day. I argued for improvement of the rules that were pushing rejects into that queue, I argued for better analysis to be completed before a rule would be added, I argued for accepting a higher risk but allowing lower value transactions through to reduce friction and make the queue more manageable. I ended up getting some of the things I was asking for, but what happened was that the rejections went down to 1-2000 per day and people just assumed now that those are fraudulent for sure since now we're "better at spotting criminals"... The introduction of another system that employed machine learning made fraud agents even more indifferent towards those queues since now "you can't fool AI". It was a very sad state of affairs and since I left there I seriously doubt that this has improved in any way. I remember the email chains you would get from some users who tried to make important purchases or even just regular purchases and not being able to get in touch with a human. I remember the frustration, the friction and the blanket statements that would emanate in team meetings saying that "it's fine as long as we protect the company from losses, so what if a few people have issues"... Those few people could have been them in other scenarios. I think this will only keep getting worse and worse with time. Each new software that attempts to bring "efficiency" to some of these types of tasks can cause a huge array of problems downstream. You end up with companies that will have 5 people working in a department that should have 500 because machine learning will "take care of the issue". I know I haven't even touched the privacy concerns that this raises, and there are PLENTY and covered in great detail in this post and in the one made by the EFF, but we're forgetting here that those affected by these things are other people. And as long as governments will say "You need to regulate this SOMEHOW" and companies will come back with "We'll use the power of AI and ML to train NN to easily identify any problems and thus bring efficiency and safety to all of our customers". We're people, we're not machines and we have a vast array of personal circumstances and elements that make us unique and saying that a blanket solution will "fix all problems" is absolutely inhuman and shows a crass misunderstanding of how these tools work and what their unintended consequences are. TL;DR - using ML/AI/NN will cause a huge array of problems. We need more people working these queues which increase exponentially so that innocent users do not get caught in your "high-tech solution". For each criminal fraud prevention rules catch, a few hundred innocent people cannot complete their activities or are labelled as criminals until proven innocent. |
What you highlighted is the reason why I wrote it. Most of the discussions are of an abstract harm against an immediate one. They other those involved.
My experience isn’t unique. It’s fairly mild.
There are people who have gone to jail because of a mistake by an algorithm. And many who have been arrested. Here’s one with Apple involved, https://www.businessinsider.com/tech/a-teen-is-suing-apple-f...
The harm is real. And I have been on both sides. I am a survivor of childhood sexual abuse. I know this would not have helped in my case. And in the cases of the other survivors I know. It was our environments that led to our experiences. The adults ignored it, were in on it, or in denial.
Apple is trying to solve a complex social problem with an algorithm. One that we are supposed to trust has been made without errors or bugs.
I can’t see the upside here.