[vkou]

They had a paper trail and processes that were documented and could be cross-examined on during discovery and lawsuits and trials.

Now it's just 'the computer says so, shrug'.

Over in the DoD, the computer says you must die, so I guess you die. Sometimes it says that about a building full of schoolchildren, but hey, nobody's at fault, the computer said so.

And it's going to get it's tentacles into every space in between. Landlord turns your application down, the computer says you are a social credit risk. Your grocery bans and trespasses you, the computer thinks you're a ne'er-do-well.

If you think none of that will happen, why not prevent it by law before it happens? Where are the hard limits of what this monster is and isn't allowed to do? How are we better off when we don't set them?

[ryandrake]

"Computer says so" has been a problem for decades too, though, just with a powerless human customer support agent in the middle. They've just automated away the computer's human voicebox.

[vkou]

Quantity has a quality all of its own. This is a paradigm shift. If it weren't, everyone wouldn't be head over heels to adopt it. It makes so many new awful uses, or turning up existing awful uses economical.

We supercharge the worst thing about computers, and you shrug and go 'eh, whatever.'

Why? Is human dignity worthless to you? Do you have no expectation of accountability from anyone with power over you or me?