[evanwise]

I can't remember where, but I saw a fairly damning argument against the intelligence explosion hypothesis on the grounds that it only works if the algorithm used to design a mind with n units of intelligence (whatever these are) scales linearly with units of intelligence. If it scales faster than linear, then your recursive bootstrapping operation takes longer and longer each time, so that eventually your next bootstrapping step will take longer than the amount of time left in the universe, meaning there is some finite intelligence cap for any such bootstrapped mind. It seems quite implausible to me that the problem of designing a mind would scale linearly, given that ostensibly much simpler problems, like sorting a list of strings, require polynomial time or log-linear time algorithms.

[rhaps0dy]

>meaning there is some finite intelligence cap for any such bootstrapped mind

This does not preclude an intelligence explosion, this cap could be many (say, 100) times higher than human intelligence. We could still see many features of an explosion in that case.

[evanwise]

True, but I think that the results of such a "weak intelligence explosion" (where the linear/sublinear scaling case would be a "strong intelligence explosion"), while still remarkable, would fall far short of some of the expectations placed on general AI by the singularity / superintelligence crowd. For the sake of argument, extrapolating from our current energy consumption using a (simplistic) linear model and some rough back of the envelope calculations, the intelligence required to harness the total energy output of the sun would be 40 trillion times the aggregate intelligence of the entire human species today. Several hundred times just isn't going to cut it. Now, if our ability to harness energy increases exponentially with intelligence, then maybe it could work, but that's just an assumption, and, given that there are hard physical limits on the efficiency of energy generation due to thermodynamics, seems very unlikely.

[jtolmar]

That's a very strange extrapolation - we don't need to be any smarter to extract more energy. We're building more renewable energy capability every year, even though humans aren't getting any smarter.

Building a dyson swarm would be a massive project, but modern humans are plenty smart enough to do it. An AI capable of running the project, while beyond the current state of the art, would still not need to be particularly clever. (It doesn't need to design satellites or space factories to get there.)

[Chickenality]

I don't understand the connection to energy consumption. Humans have been able to extract increasing amounts of energy over time without correspondingly large increases in human intelligence. I don't see a strong reason to doubt that this will continue, so I don't see a strong reason to doubt that a >= human-level-intelligence AI could do it either.

[evanwise]

The raw computing power of the human wetware has not increased appreciably over historical timescales, but the total intelligence of humanity, includes, for example, the increase in effective intelligence gained by storing knowledge in external devices like books. The gestalt organism that is "humanity" is much smarter than it was 500 or even 100 years ago, which correlates with our ability to extract resources. It's an extremely simplistic model, but since I was only after a very rough guess, I went with it.

[cbennett]

> If it scales faster than linear, then your recursive bootstrapping operation takes longer and longer each time,

Wait, does that really follow? What if you have a better than linear bootstrapping compiler. To unpack that a bit, imagine we not only have $n$ such units, but we have them wired to together in a creative way-- i dont know whether it is hierarchy, or some clever topology, but lets say that the bootstrapper now gets sub-linear scaling properties as it grows $n$.

If we look at the brain, there is a lot to be understood from the dynamics of recurrent neural fields. They are wired in a very complex way which seems to allow for some kind of very special booting (re-booting) operations. And thats just at one level of abstraction, then we re-wire them into meta-fields (like the columnar abstractions that Hawkin's builds his HTM theories around). If we have a sort of fractal information encoding, we ultimately approach shannon efficient coding. Is that what evolution has selected brains to do? And do you think it is possible the first seed AI may realize this and exploit the same strategy, just 1000x (10kx?) faster?

[nopinsight]

We should be mainly concerned with the unit of capability rather than of intelligence (for which there is no widely accepted standard measurement for non-human beings [1]) As we know, there are tasks that a less intelligent being can never accomplish no matter how much time and other resources it has.

If we use brain size as a rough proxy, ours is only three times as large as a chimp's but our capabilities for creation and destruction are vastly greater both in degree and range.

[1] IQ indicates the location in the distribution of intelligence within human population and it is flawed in many ways. The concept does not really apply to other beings.

[evanwise]

An analogous argument would hold in the case of "units of capability". Why do you think the problem of producing capable minds is any easier than the problem of producing intelligent minds? I'm not sure what the point of your chimp analogy is, unless you think I think that brain size is the unit of intelligence. Anyway, I'm also skeptical of the idea that there are such things as general "units of intelligence", but the entire notion of a superintelligence is predicated on the idea that intelligence is a general quality and can be meaningfully quantified. It's not clear to me how we'd even define superintelligence without reference to some quantitative measure of intelligence, although if you'd like to try, I'd be interested.

[mannykannot]

I think you are basically correct to say that the scalability of AGI is a speculative assumption. The argument is that if it were feasible, it might pose an existential threat, so we should consider it.

If one accepts the plausibility of AGI, then AGI that is bigger and faster than humans does not seem to be much of a stretch, but I certainly cannot imagine what capabilities a qualitatively more powerful intelligence would have, let alone how much effort would be required to get there.

Chollet claims that more powerful intelligence is, in fact, impossible, but as Scott Aaronson pointed out [1], his argument from the No Free Lunch theorem does not have any bearing on the question. As Yudkowsky points out in the article, Chollet's other arguments could just as well be used to claim that nothing beyond the level of intelligence of chimpanzees is possible.

But maybe just a bigger, faster, human-like intelligence might present a risk - people have been outsmarting one another for millennia.

[1] https://www.scottaaronson.com/blog/?p=3553

[evanwise]

Oh, don't get me wrong, AGI is still terrifying, even if it is limited to a few (or a few hundred) multiples of human intelligence by algorithmic or physical constraints. The discovery of AGI will precipitate the biggest social upheaval we've ever seen. I'm just very skeptical of the quasi-religious side of the superintelligence community, who think that we'll create godminds that can bend reality to their will or something.

Also, it didn't seem to me like Chollet was claiming that better than human intelligence was impossible, honestly. This seems to be a motivated misreading of the original article on Yudkowksy's part, and most of his article is arguing with a straw man because of it. More charitably, Chollet seems to be saying that there may be limitations to intelligence that are "built in" due to the context in which intelligence operates. For instance, even if we create a "superintelligence" in the sense that it has much greater raw processing power than humans, we may not be able to create the sensory environment and training program that would allow it to learn how to recursively improve itself without limit.

[mannykannot]

You are right about Chollet not ruling out growth altogether, but he does say - as a section heading, no less - that "our environment puts a hard limit on our individual intelligence." After a diversion into a non-sequitur about individual humans being incapable of bootstrapping their own intelligence, he launches into an extended argument that intelligence can only grow with the culture it is embedded in, and so intelligence can only grow linearly at best (linearly in what? culture?) The whole argument is a waste of time, because AI apocalypse fears are not predicated on exponential growth (and certainly not growth without limit), but only that it outstrips that of humans (and maybe not even that.) Chollet never seems to address the possibility that AGI might drive its own culture to grow, just as our ancestors' developing intelligence drove the (proto-)human culture to grow. (If Collet were to deny that this happened, then he would not be able to explain why human intelligence outstrips that of other apes, given his position that culture constrains intelligence.)