[dalbasal]

We in "the west" or whatnot need to watch out for the corporate equivalents of this, including credit scores, particularly coming from the financial sectors.

There's just a lot of opportunities these days, with all the data floating around and our increasing ability to calculate probabilities of arbitrary things about people from large enough datasets.

One side of this is disadvantage. Credit scores are an attempt to rank people by probability of default. A modern version of a credit scores would be even more black box. There are serious problems of fairness with this. You can probably calculate some usable credit score based on where you were born and to who.

Another side of such scores is the punitive side. This will affect your credit score, so stay inline. Essentially it's a lightweight proprietary justice system.

Insurance generally works in a similar way. Rental markets can too, and agents will aggregate blacklists which can then be used punitively. Policing is now using a lot more statistical techniques. Employers would probably like a commercial version of the systems police use.

Treating people as statistical objects in this way... Its dehumanising, it's discriminatory and it isn't rule of law.

[crusso]

Is it dehumanizing when I decline a ride with an Uber driver that has 3 stars?

[dalbasal]

Not necessarily, but such things can be.

Say I have a FB-based profiling system that will calculate the star rating alternative. It works slightly better than uber's current rating system. Ie, it's a better measure of how much you will like the driver. It also works before the driver takes his first fare. In fact, uber's minimum star rating policy can be used as a hiring/sign-up filter.

Would you consider such a thing dehumanising? Its essentially how insurance works, and is increasingly becoming a viable business MO.

[crusso]

Reputation systems increase our ability to trust people, products, and systems in general. It seems like our logical concern should be whether or not such systems are accurate.

The flip side of all this is that betrayal of trust is also dehumanizing. When you pay to get an Uber ride and you're taken to the wrong place or ripped off, that's dehumanizing. When you loan someone money and they don't pay you back, that's dehumanizing. When you send someone a product from an E-Bay auction and their payment turned out to be a scam, that's dehumanizing.