[icedistilled]

I mean, it's already standard practice (All states except CA) to use credit score in things like pricing your auto insurance.

It's only a tiny leap to incorporate a similar magic number some company comes up with. Actually due to competition, if it correlates to risk, all companies will literally be forced to use the magic numbers or go into an adverse selection death spiral.

Unless there is regulation against it, like in CA. Not all regulation is bad.

[TeMPOraL]

I wonder at which point it becomes a self-fulfilling property - at which point decisions based on data-driven pigeonholing actually lock people on the paths "discovered" in the numbers?

E.g. if a young adult gets classified as "disorderly, drunk, unsuitable for reproduction, suitable only for low-skill work" based on their history of college partying, and then consequently denied work and social opportunities (as everyone doing background checks sees that summary), the prediction essentially becomes a sentence.

(The third season of Westworld, despite bad writing and even worse gunfights, was very good at bringing this point up.)

[lmkg]

Read Weapons of Math Destruction by Cathy O'Neil. The book explores several ways that's already happening. Her main premise is that there's a feedback loop in many data-driven policies. You only get success results for the things that you try, and you only try the things you already think are likely to receive. As a result, algorithmic policies tend to reinforce the status quo.

Loan risk algorithms will favor people "similar to" those who have paid back loans before, a sample group biased towards people that banks have already loaned to before. As a result, a lot of the factors are biased towards "from a white upper-middle-class suburban background."

And recidivism estimators, which are used as jail sentencing guidelines in some places.

Screening algorithms for job resumes, and college applications.

Algorithms send police to where crimes are reported. Crimes are reported because the police are there to witness them. The area gets designated a high-crime area. Regular people are arrested more often because regular activity is suspicious in a high-crime area, affecting their future prospects. The higher arrest rate is used to justify this.

It's a continuous spectrum rather than a single point. But if I were to pick a single "point" where it became a self-fulling prophecy? 1994, due to the widespread passage of three-strikes laws.

[gruez]

Can't this be solved by randomly giving out the wrong prediction, and seeing how it turns out? eg. for 1% of applicants, pretend to give them 800+ credit score, then check the outcome compared to the "expected" score.

[inetknght]

You're playing with peoples' lives here. That's not funny at all.

[zozbot234]

Yup, this is what "structural" social prejudice and discrimination are all about, once you strip away all the pointless and meaningless rhetoric that's somehow supposed to be "about" these issues. It's a self-perpetuating equilibrium of basic social conditions and superstructure (viz. discourse, supporting ideas, commonly-held worldviews etc.) that create a nearly unescapable "trap" of invisible oppression.

[joejerryronnie]

A Brave New World did it easier, simply sort each person into a genetically pre-determined path prior to birth. No reason to go through all that messy data collection and analysis.

[rayhendricks]

Yes this pattern of data-driven decisions on our loves is troubling. However if we understand what these corporations are looking for it is easy to exploit them for fun and profit. If you get a decent credit score say hello to multiple $500 credit card opening balances, free plane tickets, free hotels.

[api]

This is kind of the meta plot of Gattaca. You get what you measure, so if we try to measure "potential" we end up getting only the potential our measurements say we have.

[white-flame]

One difference there is that people do have the right to view and challenge credit reports, while all this other social scoring is hidden from the targeted people.

[jacques_chester]

> Actually due to competition, if it correlates to risk, all companies will literally be forced to use the magic numbers or go into an adverse selection death spiral.

To expand on this: adverse selection is where a consumer has hidden information about the cost they can inflict on a provider. Usually this is talked about in terms of insurance (especially health insurance), but risk is risk and so the principles are the same.

My recollection of what economists predict is that there are two stable equilibria that achieve Pareto efficiency. The first is that discrimination (in the general yes/no sense) is completely forbidden and risks are totally pooled. The second is that complete discrimination is possible without limitation.

The worst outcomes are all found in attempted compromises. Not only do you have the costs of whatever tradeoff you chose between pooling and discrimination being less efficient than the Pareto points, but you also introduce a great deal of dead weight due to complex regulation and oversight, plus efforts made to evade regulation and oversight. Collectively we are worse off, even if individuals think otherwise.

I don't think allowing total discrimination is a viable option in this day and age. Which means banning it wholesale and encouraging the formation of universal risk pools.