[btilly]

In 1989, Penrose picked up Lucas' 1961 argument that no computer can possibly simulate intelligence. The argument rests on fundamental misunderstandings of logic, that are well-known among logicians. See, for example, https://www.ams.org/journals/bull/1995-32-03/S0273-0979-1995... for an article explaining this, written some 30 years ago.

The fact that Penrose has maintained his misunderstandings for 30 years, demonstrates that, on this topic, he has been a crank for a long time. No matter his other accomplishments.

[sarchertech]

Penrose essentially wrote an entire book responding to that critique.

Which side you support largely comes down to a philosophical question. The notion that he just made a stupid mistake and doubled down on it is absurd.

Having a different opinion about the soundness of a proof could make him wrong but it hardly makes him a crank.

[btilly]

The fact that he repeated the same logic errors at book length doesn't change the fact that they are errors. And the question of whether they are logic errors is a question of mathematics, not philosophy. Dismissing the conclusions of logic on the basis of philosophy, is a mistake of the same type as dismissing the conclusions of science on the basis of theology. Logic cannot speak to the philosophy that Penrose pushes forth. But it can and does speak to the validity of the argument from logic that he puts forth in support.

Hilary Putnam did a good job of explaining the mistakes. I am not a logician, but my background in logic is good enough to verify the explanation. And every logician that I personally know has come to the same conclusion.

Like you, I find it absurd to claim that Penrose has been doubling down on a basic logic error. And yet we have the basic logic error, and Penrose has clearly been doubling down on it.

You don't even need to be an expert to understand that he can't be right. Penrose argues that the capacities of human reasoning is such that Gödel's theorem proves that a mathematician's brain cannot be replicated by any mechanical process. But the reasoning process that mathematicians use is fallible. The output most emphatically is not logically consistent. The appearance of consistency is only obtained after much reexamination of those errors which were discovered. Absolute certainty of lack of error is unachievable by any kind of human reasoning. The history of mathematics is filled with examples of errors that were not discovered for shockingly long periods.

So we do not have a proof of the consistency of human reasoning, or its products. Therefore Gödel doesn't apply. Human reasoning, including the outputs that Penrose cites, do not strictly follow first order logic. Therefore Gödel again doesn't apply. And Gödel is entirely silent on the potential prospects of a heuristic algorithm that can produce inconsistent results. Which is what our brains do.

The inapplicability of Gödel's theorem to our thinking process is an absolute barrier to Penrose's attempts to prove that our thinking process cannot be the result of a mechanical system. It may be that it is not. Personally I fail to see how a strictly mechanical process can create my experience of consciousness. But this is a question that Gödel's theorem cannot address.

[sarchertech]

The mistake you are making is the same that I think a lot of people who dismiss his argument make.

You aren’t reading his actual argument. You are reading a characterization of his argument by a critic.

You can read his rebuttal of those critiques here:

https://calculemus.org/MathUniversalis/NS/10/01penrose.html

The summary is that there is no requirement that human reasoning is infallible in his actual argument.

Again his proof may be faulty. But it is not because of a “basic logic error”. The disagreements people have with his actual argument are much much subtler than a basic logic error.

[btilly]

The mistake that you are making is to imagine that I must be making a mistake.

Let's take a few examples.

He claims that a robot which is able to engage in Gödelian reasoning, cannot possibly be computable. Logicians agree that this claim is false. Indeed https://link.springer.com/article/10.1007/s10817-021-09599-8 shows a version of Gödel's theorem that has been fully checked via proof assistant. While we still lack AIs that are able to produce such proofs (other than by regurgitating such proofs in their input data), in principle a proof checker filtering the output of a brute force search through possible proof attempts will achieve any possible machine checked proofs. But proof checkers can proof check Gödelian reasoning. Thus we already know how to write a (rather impracticable) robot that does exactly what Penrose claims to be impossible.

Here's a whopper. Let's go to this passage from 4.2 of his rebuttal.

However, I had been disturbed by the possibility that there might be true mathematical propositions that were in principle inaccessible to human reason. Upon learning the true form of Gödel's theorem (in the way that Steen presented it), I was enormously gratified to hear that it asserted no such thing; for it established, instead, that the powers of human reason could not be limited to any accepted preassigned system of formalized rules. What Gödel showed was how to transcend any such system of rules, so long as those rules could themselves be trusted.

This is complete and utter bullshit. Gödel did not show that we could transcend any such system of rules. What Gödel demonstrated is what those rules can prove of themselves. Namely, "If this set of axioms proves itself consistent, then it is inconsistent." Which statement can be proven using nothing more than arithmetic. Our ability to prove this doesn't prove that our mathematical reasoning is somehow beyond what a mere formal system can prove. It is just a demonstration that we can follow a piece of arithmetic to its logical conclusion. Any other understanding of the result is simply a mistake.

He's also wrong about whether there are problems that, in principle, are beyond human reason. For example consider the BB(n) problem. Identifying which Turing machine gives us BB(643) is impossible from ZFC. (See https://github.com/CatsAreFluffy/metamath-turing-machines for more.) If you go to BB(1000), no set of axioms that mathematics has ever debated can suffice. Going beyond human comprehension doesn't take much more than that.

Of course those are weak estimates. In fact it is likely that BB(10) is going to be forever beyond us. And no, some magic quantum decoherence in the microtubules isn't going to fix that.

Let's move on. Section 4.5. He admits to the logical possibility that he is wrong, then asks whether unsoundness is plausible. How is it not plausible? The only form of intelligence that we have an existence proof for, us, thinks in notoriously unreliable ways. LLMs are our best attempt to replicate our verbal abilities by computers. They are likewise extremely unsound.

The burden of proof that soundness is possible here is on Penrose. And he needs to prove it soundly enough to overturn the generally accepted conclusion that the known laws of physics suffices, in principle, to explain the manner by which our brains operate. Because that is the conclusion that he is aiming to convince people of.

He doesn't even try. He waves his hands, declares absurdity, and moves on. That may be fine from the point of view of his philosophy. It is not fine from the point of view of a logician. It's a gap. And a mighty big one at that.

I could go on, but what's the point? If you refuse to believe what logicians say about logic, then no explanation of what logicians have to say will convince you. And if you do believe what logicians say about logic, then you should already know that Penrose is wrong.

[sarchertech]

>Humans are unsound, so Gödel doesn’t apply.

As Penrose points out, the claim is conditional. Take a recursively axiomatized theory T for which we have Pi_1 soundness level warrant, i.e., we have reason to think its Pi_1 theorems are true in the standard model N for the arithmetical cases we care about. Then we can see (again: in N) that G_T is true while T cannot prove it.

That gives principled grounds to adopt Pi_1 reflection or otherwise step to a stronger T'. No infallibility claim about people is require. This mirrors the ordinary kind of warrant mathematicians use when they adopt new axioms after scrutiny and debate.

>A proof checker plus brute-force search can do Gödel reasoning, so a robot can do it.

A checker plus search enumerates exactly the theorems of whatever fixed system it’s tied to. You can also script computable progressions that iterate reflection or consistency along recursive ordinal notations in the Turing/Feferman style. That’s still a single computable progression determined in advance. It isn’t that such progressions don’t exist, but that our justified acceptance is not a priori bounded by any one fixed computable progression.

The mechanist reply here is an existence thesis: there exists some computable procedure whose output matches everything humans could in principle come to rightly endorse for arithmetic. If that’s your view, give the existence argument. If instead you propose a specific computable progression P that we could in principle ratify as Pi_1 sound in toto, Gödel/reflection immediately pushes past P. If you say we can’t justifiably ratify P as a whole, you’ve conceded the point that no single precommitted computable scheme captures the moving boundary of what we’re warranted to accept.

>Gödel didn’t show transcendence.

In the narrow sense above, Gödel shows how to go beyond any accepted, fixed rule set once we have Pi_1 soundness level warrant for it. That is exactly the move at issue. Note the scope: inside a system you can’t adopt “if provable then true” without collapse by Löb. Outside, adopting restricted Pi_1 reflection is the justified step. If you want to reject the move, reject the warrant. Deny that we sometimes justifiably regard a given fragment of T as Pi_1 sound for the cases at hand. That’s an objection about epistemic warrant, not a logic gotcha.

>Busy Beaver shows limits, so case closed.

Busy Beaver actually helps here. It yields concrete Sigma_1 statements that can be independent of strong base theories, illustrating that warrant-driven extensions are sometimes needed to settle specific instances.

Either sometimes we do have adequate Pi_1 level meta warrant for a theory T on the arithmetical claims we care about. In which case we can recognize G_T as true in N and rightly move to T', ensuring our recognitions outrun that fixed T. Or deny such warrant altogether, in which case you haven’t refuted the conditional. You’ve just changed the target to a weaker notion of “what we can recognize.” The “humans are unsound” line doesn’t touch that conditional, and “just use a checker plus search” doesn’t answer the challenge unless you can support the existence of a single computable theory or precommitted progression that captures everything we could in principle come to rightly accept for arithmetic.

[spot]

yup and his book was reviewed as such at the time. mention of the rejection of his theory by professional philosophers however keeps getting edited out of the wikipedia page. See this exchange on the talk page https://en.wikipedia.org/wiki/Talk:The_Emperor%27s_New_Mind

>> "The book's thesis is considered erroneous by experts in the fields of philosophy, computer science, and robotics."

> Wooooah, there. That's a massive accusation to add, unsourced, and without any discussion. There needs to be a source for this statement, not to mention an opposing view. It seems unlikely the guy would win an award for a book no one thinks is right. I'm deleting it unless someone comes up with a pretty good source. Joker1189 (talk) 20:43, 27 July 2010 (UTC)

The source was provided with the edit: L.J.Landau (1997) "Penrose's Philosophical Error" ISBN 3-540-76163-2 http://www.mth.kcl.ac.uk/~llandau/Homepage/Math/penrose.html Spot (talk) 03:08, 31 July 2010 (UTC)

[sarchertech]

You say keeps getting edited out but I see one instance of that happening 15 years ago.

The talk is unavailable, but that and a chapter in Landau’s book hardly seem like appropriate sources for “everyone disagrees with him”.

The vast majority of critiques of his argument that I’ve read are by people who are actively working in AI not professional logicians.

[spot]

who said "everyone" or did you make a strawman?

https://en.wikipedia.org/wiki/Penrose%E2%80%93Lucas_argument

this actually puts it better:

The Penrose–Lucas argument about the implications of Gödel's incompleteness theorem for computational theories of human intelligence was criticized by mathematicians,[16][17][18][19] computer scientists,[20] and philosophers,[21][22][23][24][25] and the consensus among experts[7] in these fields is that the argument fails,[26][27][28] with different authors attacking different aspects of the argument.[28][29]

so, rejected by consensus. someone should update the book page so this expert rejection is clearer.

[sarchertech]

Everyone is obvious hyperbole because I didn’t want to both copying and pasting the actual quote.

That is merely a compiled list of people who disagree with him without listing any of his supporters.

There are at least 5 philosophers who support his position if you follow those links and 5 who reject it.

Link 7 doesn’t support the statement “consensus among experts in these fields” because it only refers to a single field—philosophy.

Many of those sources are just links to lists that other people have compiled of arguments for and against Lucas’ argument. They aren’t even all critiques. And many of the ones that are, are already linked directly in the article.

There’s is nothing more to support the notion that there is widespread consensus against his argument.

There may be. But this isn’t good evidence of it.

[btilly]

I can't speak to the general opinion among philosophers about his argument. But my opinion about philosophy is such that their opinion would not sway mine in either way.

I can speak to the general opinion of his logical arguments among logicians. And it is not just widespread consensus against. It is a widespread consensus that the argument is filled with basic logic errors that render it absolutely wrong.

As Hilary Putnam points out, Penrose's arguments are even worse than Lucas'. In particularly Penrose argues that no program that we can know to be sound, can simulate all our human mathematical competence. But our brains do not use a sound thinking process. Therefore a sufficiently good simulation of our brains that it can do mathematics, would also not be sound. Gödel's theorem is entirely silent on the potential capabilities of such unsound systems.

Furthermore LLMs provide a convincing demonstration that unsound simulations of us can have surprising levels of competence. ChatGPT regularly demonstrates both its competence and unsoundness. Sometimes at the same time!

The potential for unsound systems to demonstrate competence far beyond what most expected, is demonstrated by LLMs. Admittedly the current error rate is unacceptably high. But it demonstrates that what Penrose claimed to be mathematically impossible, may plausibly become real within our lifetimes. (Though, given how old Penrose is, not his.)

[sarchertech]

I replied to your other post. His proof is much subtler than that and if it has flaws that isn’t it.

https://calculemus.org/MathUniversalis/NS/10/01penrose.html

[btilly]

I am amused that my unintended repetition in the last two paragraphs demonstrates how unsound my absolutely human brain is.

(Unless I'm an LLM. I'm certainly unfunny enough to be one.)