[taneq]

Interestingly I was just today explaining the paperclip optimizer scenario to a friend who asked about the dangers of AI, including the fact that there's almost no general optimization task that doesn't (with a sufficiently long lookahead) involve taking over the world as an intermediate step.

(Obviously closed, specific tasks like "land this particular rocket safely within 15 minutes" don't always lead to this, but open ended ones like "manufacture mcguffins" or "bring about world peace" sure seem to.)

[OscarCunningham]

> "land this particular rocket safely within 15 minutes"

This one becomes especially dangerous after the 15 minutes have passed and it begins to concentrate all its attention on the paranoid scenarios where its timekeeping is wrong and 15 minutes haven't actually passed.

[taneq]

Ooh true, that could generate some interesting scenarios. "No, it's the GPS satellite clocks that are wrong, I must destroy them before they corrupt the world and cause another rocket to land at the wrong time!"

[XorNot]

Always a good time to post Jipi and the Paranoid Chip: https://vanemden.com/books/neals/jipi.html

Which pretty much tackles these issues head on.

[lancengym]

Perhaps all AI eventually figure out that humans are the REAL problems because we don't optimize, we lust and hoard and are envious and greedy - the very antithesis of resource optimization! Lol.

[taneq]

We're just optimizing (generally quite well, I might add) for genetic survival.

[NaturalPhallacy]

Ian Banks did a really amazing exposition of this where the Culture was rallying to stamp out reproducing nanites and they had to be stopped because if not they'd literally turn the whole universe into copies of themselves. One of the human characters mused that isn't that what all life is trying to do? I think it was in the Hydrogen Sonata, but I'm not sure.

[taneq]

Yes! I often find myself thinking of organisms as 'hegemonising swarms.'