Do many "otherwise smart" people actually believe "superhuman machine intelligence is prima facie ridiculous"? I'd like to see some citations :-). I think smart people tend to have much more nuanced views.
>Do many "otherwise smart" people actually believe "superhuman machine intelligence is prima facie ridiculous"?
I don't know how "otherwise smart" I am, but I wonder how we would be able to tell that a machine intelligence was "superhuman" as opposed to "buggy".
For example, suppose we build a super-AI and ask it, "Is Shinichi Mochizuki's proof of the ABC conjecture correct" [1]. What would we do if it said "yes"?
(Of course, if "superhuman" just means "able to do things humans already know how to do and verify, but lots faster", then we're already there).
What if we ask a super-AI for a proof of the ABC conjecture, and the result is something too complicated for humans to verify?
My point, if I have one, is that when I read about "superhuman machine intelligence", sometime people seem to mean "capable of knowledge that humans couldn't figure out on their own but that humans can understand once they see it"; and sometimes they seem to mean "capable of knowledge that is beyond human capacity to even verify".
I think development machine intelligence of the first kind is extremely likely, but I'm more skeptical about the second kind.
See the reaction of tech industry after Musk donated 10M USD to AI research. It sort of divided into two groups, one saying that it's great choice and another claiming that he's an idiot and AI is a hoax (for the record, I'm in the former group).
Sam's last post on Machine Intelligence, and the worries regarding it, received a lot of dismissal here on HN from people who thought that the idea is completely unfounded and implausible.
I am, by most measures, pretty smart, and I agree with Dijkstra that the question of whether a computer can think is as interesting as whether a submarine can swim.
The Strong AI hypothesis assumes a mechanistic universe, if not necessarily a materialistic one, and I think that condition is false.
I don't know how "otherwise smart" I am, but I wonder how we would be able to tell that a machine intelligence was "superhuman" as opposed to "buggy".
For example, suppose we build a super-AI and ask it, "Is Shinichi Mochizuki's proof of the ABC conjecture correct" [1]. What would we do if it said "yes"?
(Of course, if "superhuman" just means "able to do things humans already know how to do and verify, but lots faster", then we're already there).
[1] http://www.newscientist.com/article/dn26753-mathematicians-a...