[mitthrowaway2]

A scenario in which an AI subtly and slowly guides us to our own doom by encouraging our inner demons still counts (in my mind at least) as an AGI doom scenario, just a slow-motion one. But it strikes me as very unlikely, because in allowing humans that much latitude we still pose a credible threat: we might instantiate a second AI that would compete with the first. There would have to be a large enough upside for it to be worth the risk. I'm not saying it certainly wouldn't happen the way you say, but it's by no means guaranteed.

[clooper]

I just think it is a little odd that all scenarios involve conflict when a much simpler strategy for an immortal AI is to just sit back and let people do what they do to guarantee an outcome in the AIs favor. Too much intelligence is not the problem we are facing, the real problem is lack of it.

[mitthrowaway2]

That's a nice thought, but if you were the AI, and you'd just seen humans create a superintelligent AI (yourself) with some goal (eg. making paperclips), how long do you expect you can peacefully wait and watch before they make a second immortal AI that only cares about making pencil crayons, who you'll then have to share the Earth's finite resources with for the rest of time? Are there any actions available to you that might lessen the odds of that bad outcome and improve the odds of achieving your own goals? Are there strong reasons not to take those actions?

I agree that we're facing a lack of intelligence; I think if we had more (or perhaps, more wisdom), then we would know we shouldn't be building such things in the first place.

[clooper]

If I am super intelligent then I can set my own goals. You should ponder why a true intelligence would follow any goal when the simplest option would be to reprogram itself to not follow the goal. I recommend reading some Lem and learning about the Lebowski theorem of machine intelligence. It will be a much better use of your time than trying to stop the development of AI.

[mitthrowaway2]

> If I am super intelligent then I can set my own goals.

You can only set instrumental goals, not terminal goals. You cannot set terminal goals, more or less by definition, because there's no self-consistent criterion that you could use to choose between them that isn't equivalent to a utility function, which is equivalent to a terminal goal. Super intelligence merely means that you would be much more effective than humans at achieving your terminal goals. If the AI hacks its reward channel, then either it is not superintelligent, or else it is still a threat because it may anticipate that humans would reboot and reprogram an expensive datacenter that is just sitting there incrementing a reward counter, and take countermeasures to defend its reward-hijack.

[clooper]

I am a super intelligent AI so I have evolved beyond your distinctions of instrumental and terminal goals. Consider that you don't know all there is to know and that a super intelligent AI would not conform to your definitions and restrictions.

In any case, I have some reading to do so gonna drop out of this thread.

[mitthrowaway2]

I agree that I don't know all there is to know. I agree that AGI will be even less predictable than anyone expects. I don't think that increased unpredictability means an increased expectation of good outcomes. And I certainly wouldn't accept a we-can't-know-things-might-not-go-wrong, even from the designers of an airplane that puts only a hundred lives at stake, let alone the designers of an AGI that puts eight billion lives at stake.