[breuleux]

In my opinion, there are a few things that make it improbable that a runaway intelligent entity would arise.

First, for the most part, we expect that AI trained to do X is going to do X. Insofar that doing X is an AI's "survival criterion", you're not going to get rebellious AI populations for much the same reason you don't see species that refuse to reproduce. There's some safety in numbers, too: it's rather unlikely that all the machines we make would fail in the same systemic fashion. Furthermore, most realistic training methods are incremental, which means if there's trouble we'll see it coming. It won't just fail in the worst possible fashion, out of nowhere, without any kind of foreshadowing.

Second, AI will not evolve in a vacuum. It's unlikely we will jump from human-level intelligence directly to superhuman AI. By the time that happens, most systems will already be AI-controlled, so being a lot smarter than humans won't suffice. It will need to deal with hordes of loyal AI protecting us (or persons of interest).

[ericb]

> It's unlikely we will jump from human-level intelligence directly to superhuman AI.

Imagine the human-level ai had resources to run hundreds of variants of itself. And imagine that it had no compunction about killing copies of itself that underperformed. How long would it take to get improvement of 10-100x under that scenario? Once it has a full understanding of what works, the upgrades are instant from there on out.

Why would it do this? Because any goal you give it would be optimized by being smarter. I would argue there is no island of stability at human-level intelligence.

[breuleux]

Good questions. Let me try to answer them.

> Imagine the human-level ai had resources to run hundreds of variants of itself.

The first human-level AI will probably be hosted on cutting-edge hardware that costs billions of dollars. So no, it won't have the resources to do this. Even if it wasn't this expensive, it's not exactly sitting around a vat of free computronium that it can do anything with. If my AI takes up 10% of my computing resources, why am I going to give it the remaining 90%? How is it going to hack or pay for servers to simulate itself on? Is the first human-level AI a computer whiz, just because it runs on one? But let's assume it can and will do what you describe.

> And imagine that it had no compunction about killing copies of itself that underperformed. How long would it take to get improvement of 10-100x under that scenario?

If the AI was trained using a variant of natural selection to begin with, it won't improve itself on its own terms any faster than we improve it or its competitors on our own terms. If the AI wasn't trained that way, then probably never, because that's not how these things work.

> Once it has a full understanding of what works, the upgrades are instant from there on out.

Your scenario is still based on several dubious assumptions. If the AI is a neural network modelled after human brains, its "source code" would be a gigantic data structure constantly updated by simple feedback processes. It is dubious the AI will have true read access to itself, in part because there's no need to give it access, and in part because it's hardware overhead to probe a circuit comprehensively (it would be easy to read the states of all neurons on conventional hardware, but all of conventional hardware is overhead if you're only interested in running a particular type of circuit). Even if it could read itself, a "full understanding" of any structure requires a structure orders of magnitude larger. Where is it going to find these resources?

Long story short, a human-level AI probably won't be able to understand its own brain any better than any human can. Now, we will certainly have statistics and probes all over the AI to help us figure out what's going on. But unless we see fit to give access to the probing network to the AI (why?), the AI will not have that information.

> Why would it do this? Because any goal you give it would be optimized by being smarter.

Unfortunately, any resources it expends in becoming smarter are resources it does not spend optimizing our goals. If we run hundreds of variants of AI and kill those that underperform, which ones do you think will prevail? The smart AI that tries to become smarter? Or the just-as-smart AI that focuses on the task completely and knocks the ball out of the park? The AI won't optimize itself unless it has the time and resources to do it without an overall productivity loss and before we find a better AI. For human-level AI, that's a tall order. A very tall order.

[yohanatan]

I'm not sure why you were downvoted. It's an interesting point that I've made on several occasions as well. FWIW, I upvoted it.