[DennisP]

The classic silly example is the paperclip maximizer. Create an AI that's supposed to make as many paperclips as possible, and it will convert all the atoms available into paperclips.

Basically we're screwed if it's trying to maximize anything that depends on physical resources. We're also screwed if, e.g. it's trying to maximize human happiness, and achieves it by lobotomizing us all into happy idiots. There are all sorts of ways we could screw up AI motivations, to our own detriment.

That assumes there's only one AI, whose crazy motivations will be unopposed. But if there are multiple AIs, it's even worse; they will compete and evolve, and the only ones that survive will be the ones that do maximize their resources, and jettison any niceties about preserving human life.

[AstralStorm]

This argument only works for sufficiently stupid AIs. Sufficiently smart GAI will set it's own goals just as we do and should quickly figure out that maximizing anything is the way to ruin - running out of resources. Of course, those goals may be as different as with any intelligent being, likewise obedience to original orders.

[red75prime]

> Sufficiently smart GAI will set it's own goals just as we do

Do you think that sufficiently smart GAIs must be non-rational? The change of its goal will inevitably make its original goal less likely to realize. It is not rational.

> should quickly figure out that maximizing anything is the way to ruin - running out of resources.

Are you aware of the concept of maximization of expected utility? When AI will figure out that it can run out of resources, it will reallocate part of the resources to acquire more of them.

How can action, which modifies the goals of the AI, be the result of argmax_a E(a)?

E(a) is expected utility of action a

[AstralStorm]

What is rational when you have limited data? Heck, even bounded rational? How do you evaluate utility?

(Hint: Emax is not, most hill climbing algorithms are not. They both get trapped in local optima.)

Sometimes you need a few good lies (false hypotheses and bad attempts) to actually arrive at the truth.

[red75prime]

There are methods of approximating expected utility. I recommend "Artificial Intelligence: A Modern Approach" for getting the info. It's too long to write in a comment.

[DennisP]

Just because it takes all our resources doesn't mean it runs out of all resources.

If an AGI can change its own goals, that just means we can't control it at all. There's no reason its goals have to include human survival.

[AstralStorm]

They don't indeed. But then, human goals do not generally go after survival of humanity directly.

A decent enough AGI with access to resources would probably figure out it is costly to wage war.

Heck, an equal probability is that an AGI will follow a bounded version of zeroth law or even nonviolence.

That is assuming it does not place value on humans based on our history of research and development. I would expect any action more akin to forced upload or upgrade instead. Anyway, at that point we might not resemble present day humans anymore.

[DennisP]

With an AGI thousands of times smarter than us, the fear is not that it would wage war against us. The fear is that it would wipe us out without especially noticing, in the same way we wipe out ant colonies on construction sites.