Yeah, the Black Mirror reference is a stretch - Apple is rating devices, not personalities, using simple metrics. I think the mistake was them using the word “trust”, which triggers all sorts of emotional readings.
It doesn't even look like it's rating devices. It looks like it's rating behavior as it relates to past behavior.
Probably right on the usage of the word "trust". Someone probably thought "This is how much I trust that the legitimate owner is doing this" and it ran from there.
This opens up the question of whether you would rather have systems tracking trust or not. They say it's done in a way so that the score isn't identifiable by anyone at Apple but acts as a security measure to prevent fraud. In some cases, people appreciate having systems like this. It's probably better in most cases if your bank can identify fraudulent activity on your account and stop it.
There's a clear separation between bank account activity and a person, but the separation between what's done on an account attached to a phone and a person is starting to blur. We have moved so quickly toward identifying with activity on our devices that we aren't entirely comfortable with systems monitoring for fraud, but we probably also don't want those systems not to exist.
rating devices? Im sorry, I cant let you purchase that app, your screen/home button/battery has been replaced by unauthorized personnel. Please turn this device to be shredded at Apple blessed "recycling" center.
Probably right on the usage of the word "trust". Someone probably thought "This is how much I trust that the legitimate owner is doing this" and it ran from there.