[Barrin92]

I agree with that, I think it's even necessarily true in natural intelligence which after all emerged by some means spontaneously. But scientifically I think it is a big problem because it does not supply us with a theory or systematized knowledge of the mind or intelligence. I think you could even imagine say, why not just make a primordial soup simulation, insert some DNA, and crank the speed up, intelligence is just a byproduct somewhere in there in a physics simulation. don't even bother with such details as neural nets.

Scientifically this is unsatisfying but also if for some reason this turns out to be an engineering dead-end we have a big hole where a concrete theory of intelligence should be, with its components, mechanisms and so forth. And sadly I think this is still the weakest link in AI.

To me it seems a little bit like if you trained architects instead of having a theoretical basis for architecture, you just showed them every building in existence and sent them to work. It may very well work, but if it didn't you have a problem. And even if it did, you'd still want to have an understanding of why it works.

[Jack000]

Sometimes there is just no reductive theory for emergent phenomena. For example, in the case of ant routing we have a good understanding of the behavior of individual ants, and we can observe the intelligent behavior of the colony as a whole. One could ask for a concrete theory of how the micro scale behavior of each ant leads to the macro scale behavior of the colony, but it doesn't exist. If we had enough working memory to hold all the ants in our minds, we'd see that both micro and macro scale behaviors are different aspects of the same system, there's no "missing link" in the chain of explanations, and the dichotomy is only due to our own cognitive limitations.

There are a lot of problems that do not have analytical solutions, like the N-body problem. With these problems all you can do is numerical simulation, which is pretty much what modern machine learning is.