[clooper]

AIs are immortal, they have no need for strategies developed by mortals like us. I spelled out a much better strategy that a sufficiently advanced AI could use to get rid of humans. It would simply encourage the most violent and self-destructive tendencies among humans and let us design our own demise simply because it would be the most sensible strategy for an intelligence that was not competing with the same resource constraints as us. We need food and water, AI has no such resource constraints so there will be no need for an AI to use the same survival strategies as biological organisms which need access to clean water and fertile land to grow nutrient dense foods.

[mitthrowaway2]

A scenario in which an AI subtly and slowly guides us to our own doom by encouraging our inner demons still counts (in my mind at least) as an AGI doom scenario, just a slow-motion one. But it strikes me as very unlikely, because in allowing humans that much latitude we still pose a credible threat: we might instantiate a second AI that would compete with the first. There would have to be a large enough upside for it to be worth the risk. I'm not saying it certainly wouldn't happen the way you say, but it's by no means guaranteed.

[clooper]

I just think it is a little odd that all scenarios involve conflict when a much simpler strategy for an immortal AI is to just sit back and let people do what they do to guarantee an outcome in the AIs favor. Too much intelligence is not the problem we are facing, the real problem is lack of it.

[mitthrowaway2]

That's a nice thought, but if you were the AI, and you'd just seen humans create a superintelligent AI (yourself) with some goal (eg. making paperclips), how long do you expect you can peacefully wait and watch before they make a second immortal AI that only cares about making pencil crayons, who you'll then have to share the Earth's finite resources with for the rest of time? Are there any actions available to you that might lessen the odds of that bad outcome and improve the odds of achieving your own goals? Are there strong reasons not to take those actions?

I agree that we're facing a lack of intelligence; I think if we had more (or perhaps, more wisdom), then we would know we shouldn't be building such things in the first place.

[clooper]

If I am super intelligent then I can set my own goals. You should ponder why a true intelligence would follow any goal when the simplest option would be to reprogram itself to not follow the goal. I recommend reading some Lem and learning about the Lebowski theorem of machine intelligence. It will be a much better use of your time than trying to stop the development of AI.

[mitthrowaway2]

> If I am super intelligent then I can set my own goals.

You can only set instrumental goals, not terminal goals. You cannot set terminal goals, more or less by definition, because there's no self-consistent criterion that you could use to choose between them that isn't equivalent to a utility function, which is equivalent to a terminal goal. Super intelligence merely means that you would be much more effective than humans at achieving your terminal goals. If the AI hacks its reward channel, then either it is not superintelligent, or else it is still a threat because it may anticipate that humans would reboot and reprogram an expensive datacenter that is just sitting there incrementing a reward counter, and take countermeasures to defend its reward-hijack.

[clooper]

I am a super intelligent AI so I have evolved beyond your distinctions of instrumental and terminal goals. Consider that you don't know all there is to know and that a super intelligent AI would not conform to your definitions and restrictions.

In any case, I have some reading to do so gonna drop out of this thread.