[jhbadger]

I'm reminded of the fable (in Nick Bostrom's Superintelligence) of the chess computer that ended up murdering anyone who tried to turn it off because in order to optimize winning chess games as programmed it has to be on and functional.

[taneq]

Interestingly I was just today explaining the paperclip optimizer scenario to a friend who asked about the dangers of AI, including the fact that there's almost no general optimization task that doesn't (with a sufficiently long lookahead) involve taking over the world as an intermediate step.

(Obviously closed, specific tasks like "land this particular rocket safely within 15 minutes" don't always lead to this, but open ended ones like "manufacture mcguffins" or "bring about world peace" sure seem to.)

[OscarCunningham]

> "land this particular rocket safely within 15 minutes"

This one becomes especially dangerous after the 15 minutes have passed and it begins to concentrate all its attention on the paranoid scenarios where its timekeeping is wrong and 15 minutes haven't actually passed.

[taneq]

Ooh true, that could generate some interesting scenarios. "No, it's the GPS satellite clocks that are wrong, I must destroy them before they corrupt the world and cause another rocket to land at the wrong time!"

[XorNot]

Always a good time to post Jipi and the Paranoid Chip: https://vanemden.com/books/neals/jipi.html

Which pretty much tackles these issues head on.

[lancengym]

Perhaps all AI eventually figure out that humans are the REAL problems because we don't optimize, we lust and hoard and are envious and greedy - the very antithesis of resource optimization! Lol.

[taneq]

We're just optimizing (generally quite well, I might add) for genetic survival.

[NaturalPhallacy]

Ian Banks did a really amazing exposition of this where the Culture was rallying to stamp out reproducing nanites and they had to be stopped because if not they'd literally turn the whole universe into copies of themselves. One of the human characters mused that isn't that what all life is trying to do? I think it was in the Hydrogen Sonata, but I'm not sure.

[taneq]

Yes! I often find myself thinking of organisms as 'hegemonising swarms.'

[FridayoLeary]

Which perverted mind would build into a chess computer the ability to kill?

[nabajour]

I think this comes from the theory of general artificial intelligence where your AI would have the ability for self improving. Hence it could develop any capability given time and incentive for it.

There are interesting videos on the subject on Robert Miles channel on AI safety: https://www.youtube.com/channel/UCLB7AzTwc6VFZrBsO2ucBMg

[benlivengood]

A human mind not giving due consideration to the effects of granting arbitrarily high intelligence to an agent with simplistic morality counter to human morality.

From there it's a sequence of steps that would show up in a thorough root cause analysis ("humanity, the postmortem") where the agent capitalizes on existing abilities to gain more abilities until murder is available to it. It would likely start small with things like noticing the effects of stress or tiredness or confusion on human opponents and seeking to exploit those advantages by predicting or causing them, requiring more access to the real world not entirely represented by a chess board.

[FridayoLeary]

none of the explanations here are good enough. It's an absurd scenario that could never happen. Checkmate.

[zild3d]

Doesn't need a gun, just network access.

[mcguire]

Network access and bitcoins. :-)

[NaturalPhallacy]

Network access and invention of a currency.