[evanwise]

True, but I think that the results of such a "weak intelligence explosion" (where the linear/sublinear scaling case would be a "strong intelligence explosion"), while still remarkable, would fall far short of some of the expectations placed on general AI by the singularity / superintelligence crowd. For the sake of argument, extrapolating from our current energy consumption using a (simplistic) linear model and some rough back of the envelope calculations, the intelligence required to harness the total energy output of the sun would be 40 trillion times the aggregate intelligence of the entire human species today. Several hundred times just isn't going to cut it. Now, if our ability to harness energy increases exponentially with intelligence, then maybe it could work, but that's just an assumption, and, given that there are hard physical limits on the efficiency of energy generation due to thermodynamics, seems very unlikely.

[jtolmar]

That's a very strange extrapolation - we don't need to be any smarter to extract more energy. We're building more renewable energy capability every year, even though humans aren't getting any smarter.

Building a dyson swarm would be a massive project, but modern humans are plenty smart enough to do it. An AI capable of running the project, while beyond the current state of the art, would still not need to be particularly clever. (It doesn't need to design satellites or space factories to get there.)

[Chickenality]

I don't understand the connection to energy consumption. Humans have been able to extract increasing amounts of energy over time without correspondingly large increases in human intelligence. I don't see a strong reason to doubt that this will continue, so I don't see a strong reason to doubt that a >= human-level-intelligence AI could do it either.

[evanwise]

The raw computing power of the human wetware has not increased appreciably over historical timescales, but the total intelligence of humanity, includes, for example, the increase in effective intelligence gained by storing knowledge in external devices like books. The gestalt organism that is "humanity" is much smarter than it was 500 or even 100 years ago, which correlates with our ability to extract resources. It's an extremely simplistic model, but since I was only after a very rough guess, I went with it.