| HN Mirror

Exactly! This is the interesting question to ask!

And you should expect this to take a much less straight forwards form than outright malicious intend.

Machine learning stuff, to us mere mortals, is a black box that seems to output that what we desire. However, we don't need a single black box to completely hide the effect from everyone involved. It can be simply spread out over many processes without anyone noticing the sum of countless, by themselves very sensible, pro-google decisions made by google and its employees.

Say on a fuzzy edge some human or automation has to decide if a website gets banned from the index or not. Since they use adsense or google analytics it becomes possible to contact someone running the site or even, long before the offense, present them with a machine generated report suggesting decline of quality.

The google hating webmaster doesn't want all that tracking crap on his site and therefore he might be punished by not having the warning in advance. It doesn't seem harmful, it seems perfectly reasonable, if however you create 1000 such effects the meaningless 0.1% adds up to 100%.

Say, we take a user profile that use to frequent sites now banned from the search index. If that profile moves to a new website. Does that merit close monitoring? If the AI is a black box you don't even get to ask the question.

It all seems to be fair and logical but the sum of those 2 would boil down to close monitoring websites not using analytics. We would praise the AI for noticing it so soon and so efficiently.

And so things further escalate without anyone connecting the dots.

To make it more sinister:

We have a system of pro-consumerism entities that collectively indoctrinates us to enjoy consumerism. We don't know how efficient or effective that system is. But all we need to know is that we can improve it.

How hard is it really to plant a seed, water it and eat the tomatoes? How hard is it to mix flour, yeast and water then shove it in the oven? It sure seems like an infinitely complex task to me, infinite as in: ill never do it. That apathy is great news for consumerism. Its behavior that deserves to be praised and rewarded by the entities benefiting from it. There is no evil man pulling levers behind a curtain. Its just the way things are.