[AgentME]

There are non-doomsday possibilities for AGI. Imagine a super-intelligent AI that was built from the start to value humanity and our way of life, and from there it chooses to protect us and enable us all to do what we want more. (Of course, this could go in dystopian directions, but even those are better than extinction.) A super-intelligent AI that is built to value humans could decide to "uplift" humans' intelligence to be able to keep up with it in places that humans desire to keep up with it.

If we can figure out decision theory and how our values work, then when we figure out AI, we can hopefully build it to be aligned with our values from the start, instead of blindly hoping it happens to play nice with us instead of brushing us off like ants.

[nuguy]

You don’t even begin to comprehend what I’m saying. You need to think about this more deeply.

So what if it is possible to create a benevolent ai? Nobody said this isn’t possible or even likely. We can also invent a machine that scrubs all the moss off of stones. Just because it’s possible for it to exist doesn’t mean it’s going to proliferate in the free-market of the world and everything in it. The only thing that is important is the fact that

1: we will enter an unstable configuration where any AI implementation that can exist will exist

2: the AI implementations that proliferate will be those that are not hamstrung by being forced to include humans in the loop

3: humans will be out of the loop for every conceivable task and therefore not enjoy the high standard of living that they do in 2018

[AgentME]

I disagree with points 1 and 2 (and therefore 3). If a friendly AI is built and matures first, then it can protect us from unfriendly AIs trying to mature and take power. (Others call this idea a "singleton".)

[nuguy]

That’s a really good point.