[ninetyninenine]

It's definitely not like solving the halting problem. A solution 100% exists. You are it. If human intelligence can be realized in physical reality by an actual human brain, then it is provably realizable.

Few things in science exist as a north star in such abundance. We KNOW it can be built. Other futuristic things like interstellar travel... we don't actually know.

[ToValueFunfetti]

I think it maps perfectly onto the halting problem: just say one of the requirements of your program is halting. Humans can decide whether a program halts in a lot of cases, including more-or-less all of the programs we're likely to encounter. But for the overwhelming majority of possible programs, we can't figure it out.

A useful bug detector doesn't need to overcome this because it would be detecting bugs in the kind of code we write, but there is no bug detector which gives the correct answer for all inputs.

[ninetyninenine]

I don’t think you realize how universal the halting problem is in the universe.

Like the law governs everything that exists in the universe so it governs humans as well.

If a human can know that a program halts it also means the program is provably haltable. If a human doesn’t know whether a program will halt it likely means that the program is not provably haltable.

The halting problem refers to a general algorithm that can prove any program will halt.