[gaigalas]

People are so naive.

By now, everyone in tech must be familiar with the idea of Dark Patterns. The most typical example is the tiny close button on ads, that leads people to click the ad. There are tons more.

AI doesn't need to be conscious to do harm. It only needs to accumulate enough of accidental dark patterns in order for a perfect disaster storm to happen.

Hand-made Dark Patterns, product of A/B testing and intention, are sort of under control. Companies know about them, what makes them tick. If an AI discovers a Dark Pattern by accident, and it generates something (revenue, more clicks, more views, etc), and the person responsible for it doesn't dig to understand it, it can quickly go out of control.

AI doesn't need self-will, self-determination, any of that. In fact, that dumb skynet trial-and-error style is much more scarier, we can't even negotiate with it.

[Animats]

If someone sets up an AI that reads site traffic metrics and keeps trying things to increase conversion rate, something like that will happen. If someone isn't doing that already, someone will be, this year.

[gaigalas]

Dude, recommendation algorithms have been running like this for almost a decade now.

[array_key_first]

Right, but the recommendation algorithm is very contained. The YouTube recommendation algorithm doesn't write code to modify YouTube to up whatever metrics it has.

[Animats]

Of course, but now you can put the process on autopilot and let the AI keep putting dark patterns into the site by itself.