| just some thoughts on some of the requirements: > 3. This improvement is not limited by computing power, or at least not limited enough by the computing resources and energy available to the substrate of the machine. While this is a requirement, this doesn't mean that the points 4, 6 and 7 apply to the same, let's call it, generation of the AI that "escaped" from a resource limited server.
There may not even be a self improval before an unnoticed "escape". > 4. This system will have a goal that it will optimize for, and that it will not deviate from under any circumstances regardless of how intelligent it is. If the system was designed to maximize the number of marbles in the universe, the fact that it’s making itself recursively more intelligent won’t cause it to ever deviate from this simple goal. I don't see how that is a requirement. The last sentence seems to imply that deviating from the initial optimization goals automatically means the AI developed morals, and/or we don't have to worry. But I don't see any reason to believe that. > 5. This needs to happen so fast that we cannot turn it off (also known as the Foom scenario). Well, that, or it could also happen slow and gradually, but stay unnoticed. > 7. It’s possible for this machine to do the required scientific research and build the mechanisms to eliminate humanity before we can defend ourselves and before we can stop it. ... or before we notice. |