That simply moves the problem. An agent such as AIXI if it worked would start by upgrading itself to beyond what its original programmers were able to design.
Well yes it kind of does move the problem, but it moves it in an interesting direction, because it pushes it to become a self-solving problem.
The important difference is that with this approach, we could never actually get to understand how powerful AI is "really" made, as we would be isolated from it by too many levels of abstraction.
Same as currently we train deep neural networks, and we get them to work (eg. for image recognition), but we don't really know - nor care - what rules and algorithms they developed for themselves under the hoods.
Human teachers don't have to be better in every way or even more "generally intelligent" that a student in order to be helpful. Perhaps specialization of a seed AI can help breed a general purpose AIs?
The important difference is that with this approach, we could never actually get to understand how powerful AI is "really" made, as we would be isolated from it by too many levels of abstraction.
Same as currently we train deep neural networks, and we get them to work (eg. for image recognition), but we don't really know - nor care - what rules and algorithms they developed for themselves under the hoods.