| I think your premises are fair, but assumption #3 ("Any mathematical rule can be computed by a sufficiently advanced computer") is effectively ruled out by Gödel's incompleteness theorem[1] and/or the Church-Turing thesis[2]. The problem then becomes finding an approach to general AI that avoids hitting incompleteness/undecidability[3] issues. My feeling is that this would be difficult. One way to try to avoid these issues is to avoid notions of self-reference, since self-reference spawns a lot of undecidable stuff (eg, "this statement is false" is neither true nor false). It seems to me, though, that the notions of the self and self-awareness are central to human consciousness, and so unavoidable when developing a complete simulation of human consciousness. The self is probably not computable. Obviously there could be approaches that avoid these pitfalls, but every year that goes by without much progress towards general AI makes me feel more confident in this intuition. I do think there will be lots of useful progress in specialized AIs, but I see this as analogous to developing algorithms to decide the halting problem for special classes of algorithms. General AI is a whole different beast. But if general AI is physically impossible, how does the human brain "compute" general intelligence at all? It could be that your assumption #1 ("Physicalism is true. Nothing exists that is not part of the physical world.") is not correct. Maybe reality has "layers" and our world is some kind of simulation in another layer. Or maybe there is only one consciousness like many spiritual people and Boltzmann[4] suggest. Or maybe the human experience could be a process of trying to solve an undecidable problem and failing... 1. https://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_... 2. https://en.wikipedia.org/wiki/Church%E2%80%93Turing_thesis 3. https://en.wikipedia.org/wiki/Undecidable_problem 4. https://en.wikipedia.org/wiki/Boltzmann_brain |
Who says that the brain "computes" general intelligence? We don't know enough about the brain to know what it is, but it's certainly nothing like a computer. Only by analogy is intelligence something that can be computed and the only reason we have this analogy in the first place is because we have computers. But isn't the accuracy of the analogy what we would like to know with some certainty, in the first place?
This is just another big assumption that is taken for granted: that the brain is a computational device. It seems an easy assumption to make, given all we know about computation. And yet, like you say, several generations of AI researchers have failed to reproduce intelligence with computers. Perhaps the reason for this is that the brain is not a computer, intelligence is not a program, and that's why we do not BSOD when confronted with paradoxical statements like "this statement is false".