[Athanase000]

Every time something is posted from Wolfram I realize I am the only guy who likes him. Yes, he does not state, or prove, theorems. He is studying alien mathematics. Nobody has any idea of what is going on, neither does Wolfram, but he is trying! In the world of weird computability experiments he is the one with the least worst understanding of the matter. I think the fact that we have all these computational stuff laying around and we have no idea what's going on, and probably never will (by virtue of the halting theorem or something similar), contrasted with the incredible amount of understanding that normal mathematics has brought forth, is mind-boggling. You can create a turing machine from very simple rules, and nobody on earth will be able to tell what it's doing. And most machines will be like that! Most cellular automaton will be like that! Computational irreducibility is a fascinating philosophical idea. Even if it is kinda fruitless. To me it looks like there is this infinite world of meaningless computation, and then somewhere you have this happy path of actual mathematics which is full of meaning and absolutely makes sense. And somehow humans stumbled upon this happy path.

For me the ultimate proof that AI will have become super-intelligent is not that it will prove results that mathematicians can already deal with, it's that it will be able to look at a Turing machine and tell you "Yep, that's how it behaves, it follows these rules, and will never halt because this and that". I realize this is impossible in general, but if it's possible to some extent, where it is way better than the people over at busy beaver challenge, personally my mind will be blown.