[chao-]

Do those regulations really "ensure that [incidents like this] could not happen"?

I ask this in good faith, because my observation of the last few years is that the incidents still occur, with all of the harms to individuals also occurring. Then, after N number of incidents, the company pays a fine*, and the company does not necessarily make substantive changes. Superficial changes, but not always meaningful changes that would prevent future harms to individuals.

*Do these fines tend to be used to compensate the affected individuals? I am not educated on that detail, and would appreciate info from someone who is.

[gmueckl]

I don't recall the full stack of EU rwgulations in detail, but a requirement that appeal to an actual human is possible after automated decisions is in there somewhere AFAIK.

[david_allison]

> The data subject shall have the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her.

https://gdpr-info.eu/art-22-gdpr/

[sidewndr46]

But what would it matter? Wouldn't the human be an employee of the company that already made the automated decision?

[gmueckl]

A human can understand and process arguments outside the bounded input domain of automated clssification systems.

[sidewndr46]

They can, but what incentive would they have to do so? They are probably measured off the number of cases they close. The fastest way to close them would be to agree with the conclusions of the algorithm

[gmueckl]

My take on this is that telling a human reviewer to stick to a decision made by an automated process is actually against the law: some independence of the reviewer is implicitly required by the spirit of the regulation.

Naturally IANAL and such a claim would have to be tested in court if it was an actually viable argument in the first place.

[sidewndr46]

Almost certainly it is. Especially if done in writing. But it's pretty easy to do in practice. First you do it verbally, by suggesting the system rarely makes mistakes. It's the role of the employee to double check the system's work, not to second guess it obviously. Secondly just layoff or transfer anyone that doesn't side with the algorithm most of the time.

[nalak]

If I were Google I would make it a point to have the human always confirm what the AI said.

[sidewndr46]

That'd likely be a violation of some kind of laws, but you could probably work to have HR ensure that various teams were aligned in the goals of the operational attributes the company finds necessary to produce an an environment which maximizes the opportunities for individuals to contribute without fear of repression.

[badsectoracula]

Why?

[naIak]

Because humans cost a lot of money and I don’t want to train my users to think they can get a more favourable answer by asking to have a human review the decision.

[chao-]

That's a good requirement to place on these services. Thanks for educating me.

[philipwhiuk]

> Do those regulations really "ensure that [incidents like this] could not happen"?

Regulations never prevent stuff happening. They offer recompense when they do. Laws don't either.

In terms of distribution of fines, it is rare.

[CamperBob2]

Something prevented these services from originating in the EU to begin with. If not overregulation, what's responsible?

[diffeomorphism]

History, venture capital, single language market, ... . Probably a dozen different factors you could point at instead.

[Ralfp]

We've had services like that. But US competition employing thousands of people and churning billions in budgets killed them off.