[agarren]

Gödel’s incompleteness theorem, and, say, the halting problem seem to fall squarely into the bucket of “basic computability theory” in precisely the way that “we think, that is a fact”, does not (D.A. hat tip)

You’re arguing that we know artificial reasoning exists because we are capable of reasoning. This presupposes that reasoning is computable and that we ourselves reason by computation. But that’s exactly what Penrose is saying isn’t the case - you’re saying we’re walking Turing machines, we’re intelligent, so we must be able to effectively create copies of that intelligence. Penrose is saying that intelligence is poorly defined, that it requires consciousness which is poorly understood, and that we are not meat-based computers.

Your last question misses the point completely. “If we are something else, then out CoT won’t be computable…” it’s like you’re almost there but you can’t let go of “we are meat-machines, everything boils down to computation, we can cook up clones”. Except, “basic computability theory” says that’s not even wrong.