[mannykannot]

I think you are basically correct to say that the scalability of AGI is a speculative assumption. The argument is that if it were feasible, it might pose an existential threat, so we should consider it.

If one accepts the plausibility of AGI, then AGI that is bigger and faster than humans does not seem to be much of a stretch, but I certainly cannot imagine what capabilities a qualitatively more powerful intelligence would have, let alone how much effort would be required to get there.

Chollet claims that more powerful intelligence is, in fact, impossible, but as Scott Aaronson pointed out [1], his argument from the No Free Lunch theorem does not have any bearing on the question. As Yudkowsky points out in the article, Chollet's other arguments could just as well be used to claim that nothing beyond the level of intelligence of chimpanzees is possible.

But maybe just a bigger, faster, human-like intelligence might present a risk - people have been outsmarting one another for millennia.

[1] https://www.scottaaronson.com/blog/?p=3553

[evanwise]

Oh, don't get me wrong, AGI is still terrifying, even if it is limited to a few (or a few hundred) multiples of human intelligence by algorithmic or physical constraints. The discovery of AGI will precipitate the biggest social upheaval we've ever seen. I'm just very skeptical of the quasi-religious side of the superintelligence community, who think that we'll create godminds that can bend reality to their will or something.

Also, it didn't seem to me like Chollet was claiming that better than human intelligence was impossible, honestly. This seems to be a motivated misreading of the original article on Yudkowksy's part, and most of his article is arguing with a straw man because of it. More charitably, Chollet seems to be saying that there may be limitations to intelligence that are "built in" due to the context in which intelligence operates. For instance, even if we create a "superintelligence" in the sense that it has much greater raw processing power than humans, we may not be able to create the sensory environment and training program that would allow it to learn how to recursively improve itself without limit.

[mannykannot]

You are right about Chollet not ruling out growth altogether, but he does say - as a section heading, no less - that "our environment puts a hard limit on our individual intelligence." After a diversion into a non-sequitur about individual humans being incapable of bootstrapping their own intelligence, he launches into an extended argument that intelligence can only grow with the culture it is embedded in, and so intelligence can only grow linearly at best (linearly in what? culture?) The whole argument is a waste of time, because AI apocalypse fears are not predicated on exponential growth (and certainly not growth without limit), but only that it outstrips that of humans (and maybe not even that.) Chollet never seems to address the possibility that AGI might drive its own culture to grow, just as our ancestors' developing intelligence drove the (proto-)human culture to grow. (If Collet were to deny that this happened, then he would not be able to explain why human intelligence outstrips that of other apes, given his position that culture constrains intelligence.)