[zelda-mazzy]

I read their report on Rotterdam's welfare fraud prediction system a few months ago and it was fascinating. Really opened my eyes to a lot of things, specifically how fast any bias can create a feedback loop that causes one demographic to be investigated more often than others.

[nonrandomstring]

Seems two things are at play here, the model, and how the model is used. The article was quite dry, refreshingly analytical and not overly critical of the expected ground truth that the poor are punished for being poor.

It's great these models are legible and open. Looks like civil service doing a job as well as possible.

The other half of the story is how, in practice these models are interpreted and what actions follow. Are further modification of the model a direct result of that and how iterative is it?

Doing that aggressively changes a bare "model" into an investigative tool, not simply a thresholding utility. Sorry if I missed it, but I don't remember reading anything about how it feeds back. But what I did read was unsettling - entering people's homes at random, quizzing neighbours, pretty fascist stuff. am I wrong?

I guess my point is that if you look at a bare data set, or even an algorithm, sure you can probably infer a lot about biases and intent that might be built in, but you can't see the bigger model within which this functions - and that's the real story.

Is it a persecutory investigative tool?

Like Hicks said we could use cruise missiles to drop food into the mouths of hungry people... a benefit system that could identify people who are struggling (of which crime is an indicator), and, I dunno, give them some money and help? That would be preachy, and quite possible imho.

[sam_lowry_]

> a benefit system that could identify people who are struggling > (of which crime is an indicator), and, I dunno, give them some > money and help?

This is how the system should function, but then you get public outcry when the worst of us are found abusing the system of subsidies while perpetrating their crimes. I think specifically of Marc Dutroux here.

On a side note, after the mass influx of Ukrainian refugees and wide public support, it is much more obvious that those in the greatest need tend to demand less, not more.

[nonrandomstring]

Just returning a context here [0]. Interesting case of which I was unaware, Thanks.

[0] https://en.wikipedia.org/wiki/Marc_Dutroux

[VoodooJuJu]

I have limited resources for investigating robberies. I need to allocate these resources efficiently. 5x more robberies are reported in zone A than zone B. Why would I allocate an equal amount of resources to each zone?

I have limited resources for investigating fraud. I need to allocate those resources efficiently. If profile A is more likely to commit fraud than profile B, why would I allocate an equal amount of resources for investigating fraud between them?

I'm going on a hunting trip and have limited time and resources for hunting. I need to plan my trip accordingly. If deer are more abundant for hunting in Colorado, why would I go looking for them in the Sahara?

Am I wrong to look for deer in Colorado rather than the Sahara? Am I biased for investigating fraud where it's more likely to occur? Am I evil for investigating robberies where they're reported?

I have little time and resources on this planet and want to allocate my time and resources as efficiently as I can in order to do the most Good. Why would I allocate my time and resources inefficiently?

[kamma4434]

They actually do both - they spend three quarters of their time in Colorado, but they also sample random places and use that data to decide where is the next Colorado.

[Biologist123]

The model is never the reality.

[godelski]

This is often stated but I think often it is forgotten what is a model. It's not just "the algorithm." The metric is a model (the Wasserstein distance between norm layers of a discriminator is not image quality. Entropy isn't language). The data is a model (scraping all the internet isn't human language nor thoughts). It's important to always remember that these things too are models. Often proxies for things that are intractable. They are fine to use, but not fine to forget that they are not perfectly aligned. They are maps and to use another clique, the map is not the territory (no matter how detailed that map is).

[raybb]

But some are useful

https://en.m.wikipedia.org/wiki/All_models_are_wrong

[troupo]

And this is the actual danger of AI, not the doom and gloom of "AI will wipe us out".

[ilkke]

You don't need AI, just a government willing to abuse the most vulnerable. Check out the Australian "robodebt" fiasco [1]

[1]https://en.m.wikipedia.org/wiki/Robodebt_scheme

[quickthrower2]

Post Office Scandal

https://en.m.wikipedia.org/wiki/British_Post_Office_scandal

[godelski]

> You don't need AI

No one thinks this. People that are worried are worried __because__ you don't need AI. AI is just more powerful tools.

[repelsteeltje]

The thing with enforcement and AI (or more generally: automation) is that it speeds things up and amplifies biases. The silver lining of these mass profiling accidents might be that it mirrors and reveals the design flaws in law and lore that might stay under the radar when only applied individual public servants. It lifts "incidents" to "categorical failures".

[bonton89]

AI potentially adds "Computer says no" to an already bad situation. And since it is blackboxy most people can't even pick it apart when it is wrong.

[godelski]

It's mostly about scale, but there are many issues tbh

- Enables significant increase in surveillance

- Enables highly specific targeted action (i.e. micro-targeted advertising, but not necessarily ads)

- Enables said action to be performed at high speeds with low latency and for cheap.

- Obfuscates decision making processes making it harder to point to who is doing wrong and even how they are doing wrong (because burden of proof is on those being wronged. More black box = more better)

- Enables easier deflection and laziness as one can say it is just math and pretend that math is objective and not subject to usage. You can deflect claims of targeting specific groups because you do not use a variable explicitly defining those groups but you use some or all of the strongly correlating variables.

There's plenty more here too. Of course this isn't all even necessary. I do work in ML and I do have a passion for it. These tools can be great and do a lot to make human lives better. I really do think with them we are capable of doing amazing feats such as reaching a mostly post-scarce society within our lifetimes. You could have them grow the food, pick the food, transport the food, stock the shelves, be the cashiers, and even transport the people (or products) to their desired locations. We could liberate most humans of most labor. That is on the horizon. But even such an amazing outcome which would allow humans to be more human than they've ever had the ability to be in all of history -- allowing them to pursue arts, sciences, community development, and all sorts of things without needing to worry about sustaining one's survival via one's labor. The transition to this post scarce world is not just technological challenges as those jobs won't be homogeneously displaced and as we must adapt to this post world. But I'm sure everyone would absolutely love robot butlers (that are non-sentient. Sentient machines shouldn't be forced laborers, but that's a whole other conversation because sentience isn't required for this. Narrow AI can do many of these already at high competency levels). So even the purely good side is still treacherous.

I'm just saying, we need to think deep and carefully. The nuance is not just digging in the weeds, it is the critical aspect. Ignoring it isn't "good enough" when ignoring the details is what specifically leads to trouble. But it's common to hear that these are claims of pedanticism.

[pixl97]

https://en.wikipedia.org/wiki/Computers_Don%27t_Argue

[DeathArrow]

How does an investigation equal abuse? If you did no wrong, you don't have to fear any investigation about what you did with the money, and whether you deserve it or not.

By requesting welfare money, you agree that the government can investigate you.

[danielheath]

Robodebt wasn't "an investigation". It was "Our algorithm (which we know is very frequently wrong) guessed you might have money, so we've retroactively cancelled your benefits and will come after you for fraud until you can prove your innocence via an appeal process that's been deliberately obfuscated".

Governments own lawyers advised them that their plans were illegal, in advance.

[DeathArrow]

That sounds nasty. I was talking about the context about France's profiling algo.

[joshxyz]

guilty until proven innocent, yikes

[quadcore]

If you did no wrong, you don't have to fear any investigation about what you did with the money, and whether you deserve it or not.

First-hand experience here. Doesnt work like that.

How does an investigation equal abuse?

It does not in theory but in practice that is the vehicle. You have no idea what Ive been through lately via KYC in France. I was worried Id never see my money while they were making a fool out of me. It's the vehicle and if one's not a discrimination target indeed it's invisible.

[dragoncrab]

Being target of an investigation can have enormous costs.

I've been there several times.

I needed to provide ample of documents that took time to assemble and I needed to take part on in person hearings, once in a town 200 km away from my residence.

Yes, they pay for the travel and by law my employer needs to provide me extra vacation for those days, but being the guy who needs to take 2 more days again doesn't necessarily boost your career and I'm sure my family would had appreciated if I spend the weekends with them instead of reading up legalisation.

Neither case I've done anything wrong and I was always very cooperative.

I broke, when I was fined for tax evasion, presented evidence on 3 hearings that they were wrong and after they revoked the fine, I still needed to pay interest as according to law I should had paid the fine while the appeal was in progress and since I didn't, I need to pay a surcharge for the delay. Given the entire case was a huge government error I could had appeal against the surcharge, but I didn't want to go through 3 more hearings.

You can absolutely bankrupt entire families both financially and emotionally with this shit.

[pixl97]

Ok, I'm going to investigate you for child pornography as the state.

Don't worry about it when I ask your friends and coworkers about your internet browsing habits.

And oh, after all that I'll not tell anyone "Sorry, we had a mistake in the algorithm and DeathArrow had nothing to do with it at all".

[quickthrower2]

They are both arguably dangers. They are very different types of risks.

[xjay]

Indirection is the root of all manipulation.