Isn't this what we already do? We don't really understand how brains work, how what we call knowledge is stored, processed, and pruned. Psychology is a lot more immature than ML. Basically we hope that brains work well enough to define for us what "knowledge" means.
We don't understand nature, but if we're faustian, we know we can know it and that this knowledge will give us power over nature. This is something different, its a product of knowledge that we can't know or where knowledge is powerless even if possible.
Understanding nature is something a lot of people talk about, but it's not obvious to me that it is possible. As in, not possible at all, at exactly the same level as what you are referring to.
Put simply: there are a lot more possible states in a giant universe than in an individual ape's brain, or even in a collective system made up of many such apes. Wouldn't information theory tell us that we can never represent the larger system inside the smaller one? Or if we do represent it, it has to be as a model that is necessarily incomplete?
Yea, nobody is talking about knowing each trivial particular state, but general principles and processes and to on...
Knowledge doesn't imply that you need to replicate the object of knowledge to the same level of detail.