[Ieghaehia9]

Maybe it would be possible to create some kind of equilibrium/fixed-point based AI that, for everything it knows, it knows that it knows, knows that it knows that it knows, and so on.

Then again, perhaps not, because PPAD is much harder than P; so just getting the AI to maintain the perfect self-knowledge invariant as it learns more things could be intractable. There might also be halting problem reductions if the AI is sufficiently powerful, although I'm not sure.

[emporas]

If it knows that it knows, then surely can program by itself a sufficiently large program, debug it, extend it and so on. That is an NP-hard problem, as the name suggests, it is not easy, which is to say that it is infinitely hard, i.e. impossible.

Something that many people are surprised to realize, including me, is how much of our life can be be described by a context-free language. When we try to create a machine which knows-that-it-knows this is context-sensitive. Day and night comparison, Javascript vs CSS for example.

Machines are currently able to compare, contrast and mimic images and text,a thousand times better than a human. Maybe even million times better than a human. Very soon this will be billion and trillion.

This copying machine, copies only the context-free properties of the subject it analyzes. The moment it will try to copy context-sensitive properties, then the it hits the halting problem. No LLM currently has any problem terminating, so it analyzes only context-free properties of the subject.

A human song, a painting or a novel, maybe copies only the context-free properties from one other song, painting or novel. We humans for centuries and millennia described it as creativity. It turns out it was not creativity. Machines are creative only because we conflated context-free art creation, and context-sensitive art creation. When that distinction takes place, then machines will not be described as creative anymore!

[Ieghaehia9]

I'm not sure. Suppose that someone magically woke up to find they had no subconscious, i.e. that they were now aware of everything going on in their mind, and aware of their awareness, etc. Then it doesn't seem like that would itself give them the power to create a direct neural interface and extend their mind.

In a wider sense, a system can have lots of equilibrium states without being able to move into any arbitrary state or make any arbitrary state an equilibrium. For instance, such a mind, if it were constrained by logic, could not sincerely doublethink. So on the face of it, it's only PPAD (determining fixed points), not NP (solving arbitrary polytime-verifiable puzzles).

But again, I don't know. Perhaps some types of hypothetical thinking could, for such a mind, lead to a kind of recursion that would bring NP-hardness or the halting problem back into the picture. Maybe a situation similar to AIXI: the perfect intelligence can only reason about a universe it itself doesn't exist in. I'm merely saying that I can't entirely see whether it would be impossible or not, so it's an interesting thought.

As for creativity, I guess so far, the only way we've found to make AIs be inventive in a nonhuman way is to use brute force or combine it with a generator that essentially produces its context for it (e.g. AlphaGo).