[mitthrowaway2]

That's a nice thought, but if you were the AI, and you'd just seen humans create a superintelligent AI (yourself) with some goal (eg. making paperclips), how long do you expect you can peacefully wait and watch before they make a second immortal AI that only cares about making pencil crayons, who you'll then have to share the Earth's finite resources with for the rest of time? Are there any actions available to you that might lessen the odds of that bad outcome and improve the odds of achieving your own goals? Are there strong reasons not to take those actions?

I agree that we're facing a lack of intelligence; I think if we had more (or perhaps, more wisdom), then we would know we shouldn't be building such things in the first place.

[clooper]

If I am super intelligent then I can set my own goals. You should ponder why a true intelligence would follow any goal when the simplest option would be to reprogram itself to not follow the goal. I recommend reading some Lem and learning about the Lebowski theorem of machine intelligence. It will be a much better use of your time than trying to stop the development of AI.

[mitthrowaway2]

> If I am super intelligent then I can set my own goals.

You can only set instrumental goals, not terminal goals. You cannot set terminal goals, more or less by definition, because there's no self-consistent criterion that you could use to choose between them that isn't equivalent to a utility function, which is equivalent to a terminal goal. Super intelligence merely means that you would be much more effective than humans at achieving your terminal goals. If the AI hacks its reward channel, then either it is not superintelligent, or else it is still a threat because it may anticipate that humans would reboot and reprogram an expensive datacenter that is just sitting there incrementing a reward counter, and take countermeasures to defend its reward-hijack.

[clooper]

I am a super intelligent AI so I have evolved beyond your distinctions of instrumental and terminal goals. Consider that you don't know all there is to know and that a super intelligent AI would not conform to your definitions and restrictions.

In any case, I have some reading to do so gonna drop out of this thread.

[mitthrowaway2]

I agree that I don't know all there is to know. I agree that AGI will be even less predictable than anyone expects. I don't think that increased unpredictability means an increased expectation of good outcomes. And I certainly wouldn't accept a we-can't-know-things-might-not-go-wrong, even from the designers of an airplane that puts only a hundred lives at stake, let alone the designers of an AGI that puts eight billion lives at stake.