The thing with enforcement and AI (or more generally: automation) is that it speeds things up and amplifies biases. The silver lining of these mass profiling accidents might be that it mirrors and reveals the design flaws in law and lore that might stay under the radar when only applied individual public servants. It lifts "incidents" to "categorical failures".
It's mostly about scale, but there are many issues tbh
- Enables significant increase in surveillance
- Enables highly specific targeted action (i.e. micro-targeted advertising, but not necessarily ads)
- Enables said action to be performed at high speeds with low latency and for cheap.
- Obfuscates decision making processes making it harder to point to who is doing wrong and even how they are doing wrong (because burden of proof is on those being wronged. More black box = more better)
- Enables easier deflection and laziness as one can say it is just math and pretend that math is objective and not subject to usage. You can deflect claims of targeting specific groups because you do not use a variable explicitly defining those groups but you use some or all of the strongly correlating variables.
There's plenty more here too. Of course this isn't all even necessary. I do work in ML and I do have a passion for it. These tools can be great and do a lot to make human lives better. I really do think with them we are capable of doing amazing feats such as reaching a mostly post-scarce society within our lifetimes. You could have them grow the food, pick the food, transport the food, stock the shelves, be the cashiers, and even transport the people (or products) to their desired locations. We could liberate most humans of most labor. That is on the horizon. But even such an amazing outcome which would allow humans to be more human than they've ever had the ability to be in all of history -- allowing them to pursue arts, sciences, community development, and all sorts of things without needing to worry about sustaining one's survival via one's labor. The transition to this post scarce world is not just technological challenges as those jobs won't be homogeneously displaced and as we must adapt to this post world. But I'm sure everyone would absolutely love robot butlers (that are non-sentient. Sentient machines shouldn't be forced laborers, but that's a whole other conversation because sentience isn't required for this. Narrow AI can do many of these already at high competency levels). So even the purely good side is still treacherous.
I'm just saying, we need to think deep and carefully. The nuance is not just digging in the weeds, it is the critical aspect. Ignoring it isn't "good enough" when ignoring the details is what specifically leads to trouble. But it's common to hear that these are claims of pedanticism.
How does an investigation equal abuse? If you did no wrong, you don't have to fear any investigation about what you did with the money, and whether you deserve it or not.
By requesting welfare money, you agree that the government can investigate you.
Robodebt wasn't "an investigation". It was "Our algorithm (which we know is very frequently wrong) guessed you might have money, so we've retroactively cancelled your benefits and will come after you for fraud until you can prove your innocence via an appeal process that's been deliberately obfuscated".
Governments own lawyers advised them that their plans were illegal, in advance.
If you did no wrong, you don't have to fear any investigation about what you did with the money, and whether you deserve it or not.
First-hand experience here. Doesnt work like that.
How does an investigation equal abuse?
It does not in theory but in practice that is the vehicle. You have no idea what Ive been through lately via KYC in France. I was worried Id never see my money while they were making a fool out of me. It's the vehicle and if one's not a discrimination target indeed it's invisible.
Being target of an investigation can have enormous costs.
I've been there several times.
I needed to provide ample of documents that took time to assemble and I needed to take part on in person hearings, once in a town 200 km away from my residence.
Yes, they pay for the travel and by law my employer needs to provide me extra vacation for those days, but being the guy who needs to take 2 more days again doesn't necessarily boost your career and I'm sure my family would had appreciated if I spend the weekends with them instead of reading up legalisation.
Neither case I've done anything wrong and I was always very cooperative.
I broke, when I was fined for tax evasion, presented evidence on 3 hearings that they were wrong and after they revoked the fine, I still needed to pay interest as according to law I should had paid the fine while the appeal was in progress and since I didn't, I need to pay a surcharge for the delay.
Given the entire case was a huge government error I could had appeal against the surcharge, but I didn't want to go through 3 more hearings.
You can absolutely bankrupt entire families both financially and emotionally with this shit.
[1]https://en.m.wikipedia.org/wiki/Robodebt_scheme