[TeMPOraL]

I wonder at which point it becomes a self-fulfilling property - at which point decisions based on data-driven pigeonholing actually lock people on the paths "discovered" in the numbers?

E.g. if a young adult gets classified as "disorderly, drunk, unsuitable for reproduction, suitable only for low-skill work" based on their history of college partying, and then consequently denied work and social opportunities (as everyone doing background checks sees that summary), the prediction essentially becomes a sentence.

(The third season of Westworld, despite bad writing and even worse gunfights, was very good at bringing this point up.)

[lmkg]

Read Weapons of Math Destruction by Cathy O'Neil. The book explores several ways that's already happening. Her main premise is that there's a feedback loop in many data-driven policies. You only get success results for the things that you try, and you only try the things you already think are likely to receive. As a result, algorithmic policies tend to reinforce the status quo.

Loan risk algorithms will favor people "similar to" those who have paid back loans before, a sample group biased towards people that banks have already loaned to before. As a result, a lot of the factors are biased towards "from a white upper-middle-class suburban background."

And recidivism estimators, which are used as jail sentencing guidelines in some places.

Screening algorithms for job resumes, and college applications.

Algorithms send police to where crimes are reported. Crimes are reported because the police are there to witness them. The area gets designated a high-crime area. Regular people are arrested more often because regular activity is suspicious in a high-crime area, affecting their future prospects. The higher arrest rate is used to justify this.

It's a continuous spectrum rather than a single point. But if I were to pick a single "point" where it became a self-fulling prophecy? 1994, due to the widespread passage of three-strikes laws.

[gruez]

Can't this be solved by randomly giving out the wrong prediction, and seeing how it turns out? eg. for 1% of applicants, pretend to give them 800+ credit score, then check the outcome compared to the "expected" score.

[inetknght]

You're playing with peoples' lives here. That's not funny at all.

[zozbot234]

Yup, this is what "structural" social prejudice and discrimination are all about, once you strip away all the pointless and meaningless rhetoric that's somehow supposed to be "about" these issues. It's a self-perpetuating equilibrium of basic social conditions and superstructure (viz. discourse, supporting ideas, commonly-held worldviews etc.) that create a nearly unescapable "trap" of invisible oppression.

[joejerryronnie]

A Brave New World did it easier, simply sort each person into a genetically pre-determined path prior to birth. No reason to go through all that messy data collection and analysis.

[rayhendricks]

Yes this pattern of data-driven decisions on our loves is troubling. However if we understand what these corporations are looking for it is easy to exploit them for fun and profit. If you get a decent credit score say hello to multiple $500 credit card opening balances, free plane tickets, free hotels.

[api]

This is kind of the meta plot of Gattaca. You get what you measure, so if we try to measure "potential" we end up getting only the potential our measurements say we have.