| > Worrying about this stuff is like worrying about overpopulation on Mars, and there's maybe need for one or two people in the world to work on this. When something is shown to be doable in principle, it's often not clear how difficult it will be in practice. In 1932, Ernest Rutherford thought nuclear energy would not be a viable source of energy, let alone weaponized. In 1933, Leo Szilard filed a patent on the concept of the neutron-induced nuclear chain reaction. At this point in time, nuclear fission was not yet known and actually making nuclear energy viable was a pipe dream. As we all know, in 1945, the first nuclear weapons were used. Until the day that the weapons were used, German physicists thought that nuclear weapons would not be used in the war because, while possible in principle, the actual construction of a working device would require a herculean effort that no nation would expend in time for the war. The German physicists weren't too far off in estimating how difficult nuclear weapons were. They just failed to predict that the US would throw 130,000 people, including most of their top minds, at the problem for years. Now, we have no idea how difficult superintelligence will be. But the possibility that we're a couple of breakthroughs and a Manhattan project away from superintelligence is real, and I want a hell of a lot more than one philosopher and an eccentric fanfic writer working on this. EDIT: No offense to Yudkowsky. I thought the fanfic was fairly good and, more importantly, achieved its purpose. |