| We know that for most problems, a group of people tends to be better at problem solving than an individual [1]. Even if AI technology only reaches human-level and does not exceed it, continually increasing efficiency would make AI smarter than any small groups of humans and immense bandwidth relative to human communications would make it more effective than any large human organizations. [1] https://aiimpacts.org/coordinated-human-action-example-super... In addition, if AI posesses sufficient computing resources, which will certainly become available in the next few decades if not already, it will have inherent strong advantages like serial computation speed and memory size that exceed any human brains. So the real barrier for AGI is software and not hardware. If AGI software is developed before we have sufficient hardware to run it at the human brain speed, then it will become more capable at the rate of hardware we can put into use, which is likely exponential given how parallelized the human brain appears to be. The major counterargument I find most convincing regarding outsized impact of exploding intelligence is that many problems are exponentially hard (or harder than that) and thus exponential intelligence can only make linear or sublinear progress on them. However, linear or even sublinear progress may still lead to quite drastic changes in the world. If an organization can marginally predict stock price movements better than the rest of the world, in a few decades it will accumulate great resources and power. The same is true for many other important domains. |
> the real barrier for AGI is software and not hardware.
This is indeed true. It has actually been true for some time. AGI is computable on present day hardware if one has enough knowledge on how to correctly structure it. A fundamental understanding of intelligence is the first step. How one crafts this understanding into software is the second step. Hardware reached capability in recent years.
> exploding intelligence
There is no such thing. Time is still required like with all things. Teaching/learning/interaction is still required. Furthermore, A controller/overseer of the system can more than adequately limit progress they are not comfortable with. I find the idea of exploding intelligence/overnight super AI to be pure fantasy not at all aware as to the structure of AGI.
> If an organization can marginally predict stock price movements better than the rest of the world..
The problem is this kind of thinking... AGI is achieved and people rush to apply it to games to get rich. Sorry, that will not occur. It will not occur because the stock market is fundamentally a [game]. A game with disadvantaged players. A game with incomplete information. A game whose rules/dynamics change frequently to suit inside players. One could make all of the accurate predictions they wanted, if the game changes underneath you or before you can act, your lofty predictions have no real world value and that's exactly how the market behaves.