[YeGoblynQueenne]

Without getting too technical, if AI must run on a computer, and there are programs that computers cannot compute, then there are limits to what AI can compute, or in other words think (because it's an AI; so to think, it computes).

As a very crude example of this kind of limitation, I'm sure I've read a hundred Sci-Fi stories where the humans defeat the AI by posing to it an unsolvable problem, usually a variation of the "liar's paradox", e.g. "this statement is false" (which btw is used by Gödel in his proof). So a human traipses over to the superintelligent AI and blurts "I always lie". Then the AI spends an aeon processing the sentence and then blows up.

Even worse, intelligence itself may turn out to be an uncomputable program that cannot be executed on a computer. In that case- no AI.

[visarga]

The scenarios you described are not realistic. There's no AI getting stuck into a loop by a tricky question, not even GPT-3. Any solution would take into consideration its computation cost. AlphaGO for example would evaluate 50K board states per move, it won't go into a 3^361 recursion.

In general when the problem is so hard evolutionary methods are suitable. They naturally blend the notion of cost with that of search and cope better with deceptive objectives.

[YeGoblynQueenne]

Your turn of phrase "there's no AI getting stuck..." has me stuck. Are you assuming that there exist "AIs", as in artificially intelligent entities, like the kind imagined by science fiction writers and (some) AI researchers, alike? To clarify, there are no such systems. "AI" is the name of a research field, not any ability that characterises a class of systems currently known.

This suffices to explain why there is, indeed "no AI getting stuck inot a loop by a tricky question". Because there is "no AI" at all, certainly not of the kind that can understand a "tricky question" sufficiently well to stumble on the paradox inside it.

For example, GPT-3 has no ability to process "this sentence is false" in such a way as to decide its truth. AlphaGo is not capable of processing language at all, it is only capable of playing board games and it isn't even capable of playing board games by reasoning, only by search. AlphaGo searches a game tree structured as a directed graph, without loops so it's hard to see how it could get stuck on recursive paradoxes anyway.

In general, such systems as exist today do not have the mathematical properties of the formal systems described by Gödel, Church and Turing. They don't even have memories. So they are, let's say immune to incompleteness, because they're not even incomplete.

[simiones]

It's important though to note that we know of no problem so far that a human mind can solve that couldn't be solved by a Turing Machine. Godel's incompleteness theorems may very well apply to human minds as well, and so far this seems more likely than not (though a lot of human thinking is actually inconsistent, so perhaps it falls to the other side of the incomplete/inconsistent "choice" than formal systems).

[YeGoblynQueenne]

Well... maybe human minds are themselves as limited as Turing machines? In that case we may never be able to create a machine that can overcome our, and Turing machines', limitations.

[simiones]

That is exactly what I personally believe.