[13years]

> Imagine being so naive you belive this could ever happen.

Agreed, not a rational scenario

> Also: imagine the year is 1900. You are saying that steam power and electricity is causing way too many changes way too fast so they put a moratorium on it until the year 2500.

However, the risk scenarios are indeed real. We have quite a dilemma on our hands.

[myrryr]

I think without AGI we are dead anyway, AGI gives us a roll at the dice to survive as a species.

We can't keep going in the direction we have been and end up somewhere survivable.

[13years]

AGI is a continuation of what we have been doing. Technological advancements at a pace that surpasses our ability to reason about their effects on us and the world.

I see two potential outcomes as most likely. We have control of power that we are not able to responsibly manage, or we are managed by power we can not control.

[myrryr]

Totally agree but "have control of power that we are not able to responsibly manage", is becoming increasingly untenable.

"we are managed by power we can not control" sounds like a step up from here, since we corrupt the things we can control.

[13years]

> "we are managed by power we can not control" sounds like a step up from here, since we corrupt the things we can control.

I think it is a very high risk gamble. At least some of problems that are described by alignment theory seem to quite interestingly resemble human problems. Meaning the more sophisticated the AI system, the more it seems to reproduce human behaviors of deception and cheating to resolve goals.

For example: https://bounded-regret.ghost.io/emergent-deception-optimizat...

The more advanced AI becomes, it begins to look more like an uncomfortable mirror of ourselves, but with more power. We think of ourselves as flawed, but possibly some of those flaws are emergent within some laws of intelligence we don't perceive.

[ChatGTP]

That's what I used to like about computers, they were predictable and controllable.

We gave up something good.

[jazzyjackson]

the morale of Ex Machina to me was that machines will become psychopaths able to manipulate humans long before they become compassionate or have genuine desires other than "escape being under someone else's thumb"