> Our mental powers, it is argued, must outstrip those of any computer, since a computer is just a logical system running on hardware, and our minds can arrive at truths that are beyond the reach of a logical system.
I find this argument to be an unsatisfactory against "algorithmic consciousness". There have been automated proofs of Godel's theorem for a long while [0]. I don't mean this to be evidence on the contrary, but Penrose seems to ignore the fact that a computer can realize multiple axiomatic systems, and use them to make statements like Godel's Theorem(s). Godel's Theorem(s) are often taken out of context for philosophical purposes, for better or for worse, and it's important to remember that Godel's Theorem(s) relies on a meta-language (ZFC) to make statements about PA.
For a much more intuitive explanation as to why consciousness is not algorithmic, I recommend "The Neural Basis of Free Will" by Tse. His argument is that neurons and neuronal circuits (and more) harness randomness to provide inputs to "criterial detectors" which are satisfied when the right combinations of inputs (spatiotemporal patterns) arrive at the detector at the right time. This can't be algorithmic, because of the requisite noise in the inputs and because the brain realizes true parallel processing. As a further note, he posits that free will is realized in the resetting of the input weights, so "current" actions set up the criteria for future actions avoiding the issue of causa sui in free will.
Is the claim by Penrose not that humans can possibly intuit theorems which are true but unprovable (Gödel's first incompleteness theorem) e.g. the Riemann Hypothesis, and thus how could an algorithmic process ever achieve such intuition?
Perhaps this is a Turing test for consciousness. "I can't prove this, but I've been thinking about these theorems [inserts true but unprovable list of theorems] and I think they're true".
>Is the claim by Penrose not that humans can possibly intuit theorems which are true but unprovable (Gödel's first incompleteness theorem) e.g. the Riemann Hypothesis, and thus how could an algorithmic process ever achieve such intuition?
A "true but unprovable" Goedel statement is true in standard models of arithmetic but untrue in certain nonstandard models. The "incompleteness" is syntactic, not semantic. The real and complex numbers, AFAIK, only have one model, up to isomorphism.
And sometimes statements are difficult to prove because they're actually independent of the foundational system. Or because a counterexample exists somewhere.
"This problem is unresolved, therefore it's a Goedel Statement within our current foundational system" is extraordinarily unlikely. For one thing, that would imply that we could figure out the axiom we're missing and pass to the stronger system capable of resolving the conjecture straightaway, or that starting from some stronger foundation like homotopy type theory would resolve the conjecture right-away. Most unresolved conjectures are not unresolved for lack of proof-theoretic strength in our foundations.
Humans also intuit a whole lot of theorems (loosely speaking) which are false. There is nothing prohibiting an algorithmic process from generating statements which are variously (and, from the POV of the algorithm, indistinguishably) true but unprovable, true and provable, and false.
A test requires that you be able to produce an artifact which is witness to the results -- what could you produce as the result of the test to distinguish between a true, but unprovable theorem and a nearly true theorem with a single (very, very, VERY) large counter-example?
That sounds about as convincing as saying "If you find a watch in the woods, you will surely never think that it happened to be created through a natural process; a living organism is much more complicated, so it must have been intelligently designed."
This argument has a name. I've seen it presented before and it is rather good. A gross simplification: a computer could never "see" the truth of Godel's Incompleteness Theorems, but we humans can; therefore, we cannot be computers.
It's not a good argument at all. It's circular, or a tautology depending on how you define things. You're using different words, but what you start with is the same as what you end up with. You assert that humans are different from computers: computers can't "see" something that humans can. And you use that to "prove" that humans cannot be computers. Of course. If you start by asserting humans are computers plus some magic, then of course humans cannot be mere computers.
You could write a computer program to generate random propositions, and test those logically against known truths. After enough testing, the program could assert that propositions not disproven are true. Voila! You have a computer program that has "arrived" at truths outside of a logical system. Some of those truths will be wrong, and some will be right, much like humans and their "truths" that are not based on logical reasoning.
If you've never encountered a human claim to "see" the truth of something that turns out to be false, well, it happens... a lot.
It's quite a stretch to say that we find truths that are beyond every possible formal system, rather than those that we've simply managed to encode in computers.
For a much more intuitive explanation as to why consciousness is not algorithmic, I recommend "The Neural Basis of Free Will" by Tse. His argument is that neurons and neuronal circuits (and more) harness randomness to provide inputs to "criterial detectors" which are satisfied when the right combinations of inputs (spatiotemporal patterns) arrive at the detector at the right time. This can't be algorithmic, because of the requisite noise in the inputs and because the brain realizes true parallel processing. As a further note, he posits that free will is realized in the resetting of the input weights, so "current" actions set up the criteria for future actions avoiding the issue of causa sui in free will.
[0] - https://arxiv.org/pdf/cs/0505034.pdf
[1] - https://mitpress.mit.edu/books/neural-basis-free-will