[fennecfoxen]

The Terminator scenario is basically one, single centralized AI wired into everything (or at least wired into enough military power to stave off opposition as it force-wires itself into all the rest of the things). But practically speaking, a single centralized AI is highly unlikely to be a world-takeover machine of godlike intelligence in its first iteration. The iterative improvement that you refer to which could potentially make it dangerous also points to a likelihood of there being a diversity of agents with a variety of designs being run by a diverse set of organizations with a diverse set of goals, with many incentives to obtain protection from hostile AIs and computer systems run by 1337 h4x0rz (possibly foreign government hackers).

Diversity and independence of AI Agents mitigates against the Terminator-scenario danger; the situation would really be not so different than the modern-day situation with natural intelligences that have control over resources and weapons of war.

[Houshalter]

>Diversity and independence of AI Agents mitigates against the Terminator-scenario danger

I don't see how this is the case. The AI's still are likely to be more similar to each other than they are to humans. And they could choose to cooperate to kill humans, and then split the resources of the world among themselves.

Having multiple AIs is no guarantee of safety! Unless those AIs want to keep humans alive and care about our future.

The second assumption is that the world of the future will be stable. It could be that a small technological advantage is enough to win against every other group. E.g. if your AI can invent better nanotechnology first, or discovers some entirely unknown technology. It's unlikely that a random set of superintelligences would be perfectly evenly matched. Some would be much larger, smarter, or have access to more resources, or have a head start in time.

Lastly and most importantly, is the idea that superintelligence won't be built quickly. Once you have the first smarter than human AIs, it may be only a matter of weeks or days before you have superintelligence. That's the whole idea of an "intelligence explosion". The smarter than human AI's can do AI research and improve themselves, and then improve themselves even more, then even more. Not to mention plugging more GPUs into their brains to make them smarter.

So it seems very likely to me that once you have a lab that solves the last technical problems in creating AGI, a single superintelligence will rapidly emerge from that. The superintelligence will then be able to get whatever it wants, because it's so much smarter than humans. It doesn't need to be "wired into" anything if it's 10x smarter and faster than the best human hackers. And if it's 2050 and everything and your toothbrush is connected to the internet.

[pka]

I'm not qualified enough to comment on the possibility of strong AI, but one thing to consider is that natural intelligences don't launch the missiles only because they are afraid of the resulting total annihilation, of death basically. Maybe globally distributed strong AIs won't have such concerns.

[gm-conspiracy]

I am not so worried about explicit physical violence, but subtle manipulation of what us humans consider reality.

Also, are you assuming the iterative development requires human intervention?

[TheOtherHobbes]

If anyone is really worried about magic AI, they can build it inside a VR bottle and add failsafes to protect against god-like supersentience.

I think god-like supersentience is ludicrously unlikely, if only because it would probably have to be NP complete and bug free, and both seem like a very tall order - you immediately run into the barber's paradox where you're expecting a system to understand itself completely using its own system.

Is this likely? I'd say no. In fact I conjecture it's physically and metaphysically impossible.

More mundane super-AIs are possible, but they reduce to the out of control machine problem.If there's a problem, it will be because of the possible destructiveness of the implementation technology - e.g. nano, or even traditional bio - and not inherently because of AI.

Bio-like systems may turn out to be more efficient than silicon, so we're far more likely to have a problem if we build a smart machine self-evolving machine out of DNA than out of GPUs.

Arguably we're a smart self-evolving machine already, and the jury is still out on whether or not we're a good idea.

[Houshalter]

>In fact I conjecture it's physically and metaphysically impossible.

That's crazy. Human brains were created by nothing more than random mutations and selection. A very stupid algorithm created the most intelligent thing that currently exists. Humans are a thousand times smarter than evolution. We can invent things evolution never could.

And now humans are learning about how our own brains work through studying them, and also how to build intelligent algorithms ourselves. And we are doing it much faster than evolution did (10's of years vs millions.)

>the barber's paradox where you're expecting a system to understand itself completely using its own system.

I don't see how this is a paradox. There's no reason a system can't understand itself. The algorithms in your brain are probably much simpler than the amount of space your brain has to store information.

>Bio-like systems may turn out to be more efficient than silicon

Silicon is already vastly more efficient. Transistors can be the width of atoms, while synapses are the widths of many molecules. Neurons have to maintain all the machinery necessary for life and self replication. They use slow and inefficient chemical signals and reactions. Etc, etc.

[JumpCrisscross]

> If anyone is really worried about magic AI, they can build it inside a VR bottle and add failsafes to protect against god-like supersentience

This is sort of like a dog trying to trap its human while it raids the pantry. You have to assume, whether through technical superiority or deception, the AI will figure out how to escape (re-read Lord of the Rings imagining the One Ring as an AI).

[Houshalter]

It also assumes that humans would want to trap it. If you wanted to do that, why would you build it in the first place? The people that build the first AIs will be those like the parent commentator, that don't take the risk seriously.

[gm-conspiracy]

My concern is that the builders and the worriers are not the same entity.

[rjsw]

What if the AI just thinks that it understands itself ?

[fennecfoxen]

> Also, are you assuming the iterative development requires human intervention?

At first, yes; long-term, it doesn't matter too much. If IBM's AI agents come up with better versions of themselves they'll still be trying to rent them out to different people in order to compete with Google's and Baidu's and Lockheed's and Siemens' businesses in those market (or whoever's still around and actually running these things at that point).

[gm-conspiracy]

So, multiple benevolent feral AIs that coexist in harmony?