[stoniejohnson]

I agree the premise of FOOM is very unlikely.

But if you assume that we have created something that is agentic and can reason much faster and more effectively than us, then us dying out seems very likely.

It will have goals different from ours, since it isn't us, and the idea that they will all be congruent with our homeostasis needs evidence.

If you simply assume:

1. it will have different goals (because it's not us)

2. it can achieve said goals despite our protests (it's smarter by assumption)

3. some goals will be in conflict with our homeostasis (we would share resources due to our shared location, Earth)

then we all die.

I just think this is silly because of the assumption that we can create some sort of ASI, not because of the syllogism that follows.

(As an intuition pump, we can hold on the order of ones of things in our working memory. Imagine facing a foe who can hold on the order of thousands of things when deciding in real time, or even millions.)

[card_zero]

Point 1 is a big assumption. I am also not you, and although it's true that I have different goals, I share most of your human moral values and wish you no specific harm.

I'm also unconvinced by the idea that rapid reasoning can reach meaningful results, without a suitably rapid real world environment to play with. Imagine a human, or 8 billion humans if you like, cut off from external physical reality, like brains in jars, but with their lives extended for a really long time so that they can have a really good long think. Let them talk to one another for a thousand years, even, let them simulate things on computers, but don't allow them any contact with anything more physical. Do they emerge from this with a brilliant plan about what to do next? Do they create genius ideas appropriate for the year 3000? Or are they mostly just disoriented and weird?

[stoniejohnson]

I didn't mention it but I fully agree, I imagine ASI would have be to embodied.

My reasoning is simple, there are a whole class of problems that require embodiment, and I assume ASI would be able to solve those problems.

Regarding

> Point 1 is a big assumption. I am also not you, and although it's true that I have different goals, I share most of your human moral values and wish you no specific harm.

Yeah I also agree this a huge assumption. Why do I make that assumption? Well, to achieve cognition far beyond ours, they would have to be different from us by definition.

Maybe morals/virtues emerge as you become smarter, but I feel like that shouldn't be the null hypothesis here. This is entirely vibes based, I don't have a syllogism for it.

[card_zero]

Smarts = ideas, and the available ideas are ours, which contain our values. Where's it going to learn its morality from?

* No values at all = no motivation to even move.

* Some ad hoc home-spun jungle morality like a feral child - in that case, it would lack heuristics in general and wouldn't be so smart. Even your artificial super-brain has to do the "standing on the shoulders of giants" thing.

* It gets its moral ideas from a malevolent alien or axe murderer - how come? Unless it was personally trained and nurtured by Mark Zuckerberg I don't see why this would happen.

Mind you, I suppose it's true that even normal humans tend to be somewhat mean and aloof to any outgroup of slightly different humans. So even after it learns our values, there's a definite risk of it being snotty to us.

[jfoster]

The "different goals than us" part is redundant. Across humanity there is already vastly oppositional goals. That's why we have prisons, war, and counter-terrorist organizations.

As Geoffrey Hinton points out, a generally useful subgoal of any task is power accumulation. In other words, you can assume that a very intelligent AI will always be not just smarter than us but also accumulate power for anything that you ask it to do, simply in order to do that thing more effectively.

Imagine if everyone had access to a magic genie. Eventually someone is going to wish for something bad.

[bamboozled]

Where I think Hinton’s views fall down is that the e have zero idea of what AGI smarter than us might want or what it might do. Like people always talk about it as if an entity like that would just hang around and bully our species. It might evolve into a ball of light and leave the planet. I don’t know but we seem to assign a lot of human traits to something that wild likely be completely unrecognisable to us in probably twenty minutes after birth.

[jfoster]

But what you're missing is that he's not talking about the "desires" of an AGI.

He's talking about a purely logical subgoal in what humans might ask a very capable AI to do.

[bamboozled]

What's the difference between desires and goals in this context really? You could say he is worried about a reasoning machine "relentlessly programmed" to achieve some goal, but a reasoning machine might just reason itself out of everything you've told it to do. Something so creative, so capable, so beyond us, yet it's going to...assassinate other people for you? Why?

When something goes from being a computer program to a self-aware, conscious being with agency, things change a lot.

Hinton is a major paradox of a human, he has spent his life building the very thing he says will likely doom us, and now spends his life warning us against his own work? So much of this AI doomerism just seems like a "chinese finger trap" for the ultra logical thinker.

It's a fucking weird time to be alive. The 90s felt much less weird and dystopian to me.

[stoniejohnson]

Yeah that tracks, I just feel like the difference required to be an ASI might allow for a stronger claim here.