[xyzzy123]

You're right of course, LLMs provide a partial, unsound oracle.

The "halting problem is unsolvable" argument relies on the oracle not being able to output "not sure". But adding that option admits trivial oracles, like ones which output "not sure" for everything, so some are better than others.

The "real world" use most people have for halting oracles is as part of software safety, where if the checker outputs "not sure" you modify the software until the checker can decide if it halts.

[skybrian]

Even if we had an answer to the halting problem, from a practical point of view, it's solving the wrong problem anyway. There's no point in distinguishing a function that would run forever from one that would exit after a century. They are both too slow.

Knowing that functions terminate is important for proof languages like Lean, where you often want to prove things without running the code at all. You're proving that one could, theoretically, calculate an answer, without actually calculating it.

[fragmede]

It's an engineering vs science question, and we've solved the practical but not the theoretical. LLMs being non-deterministic is another one of those things.