[jfoster]

The "different goals than us" part is redundant. Across humanity there is already vastly oppositional goals. That's why we have prisons, war, and counter-terrorist organizations.

As Geoffrey Hinton points out, a generally useful subgoal of any task is power accumulation. In other words, you can assume that a very intelligent AI will always be not just smarter than us but also accumulate power for anything that you ask it to do, simply in order to do that thing more effectively.

Imagine if everyone had access to a magic genie. Eventually someone is going to wish for something bad.

[bamboozled]

Where I think Hinton’s views fall down is that the e have zero idea of what AGI smarter than us might want or what it might do. Like people always talk about it as if an entity like that would just hang around and bully our species. It might evolve into a ball of light and leave the planet. I don’t know but we seem to assign a lot of human traits to something that wild likely be completely unrecognisable to us in probably twenty minutes after birth.

[jfoster]

But what you're missing is that he's not talking about the "desires" of an AGI.

He's talking about a purely logical subgoal in what humans might ask a very capable AI to do.

[bamboozled]

What's the difference between desires and goals in this context really? You could say he is worried about a reasoning machine "relentlessly programmed" to achieve some goal, but a reasoning machine might just reason itself out of everything you've told it to do. Something so creative, so capable, so beyond us, yet it's going to...assassinate other people for you? Why?

When something goes from being a computer program to a self-aware, conscious being with agency, things change a lot.

Hinton is a major paradox of a human, he has spent his life building the very thing he says will likely doom us, and now spends his life warning us against his own work? So much of this AI doomerism just seems like a "chinese finger trap" for the ultra logical thinker.

It's a fucking weird time to be alive. The 90s felt much less weird and dystopian to me.

[stoniejohnson]

Yeah that tracks, I just feel like the difference required to be an ASI might allow for a stronger claim here.