There's a lot of stuff, language for example, that's mostly genetic. Humans aren't learning these things from scratch, generally they're just filling in some blanks and helping guide the development of the brain.
I generally agree, but I think there is something else going on too. Conceptualization is highly flexible and comes in really early. Little kids ask "why?" a lot. Why do they? There must be some energy minimization reason for it. I don't think we're going to get AGI without it being able to ask some questions and get some answers that fundamentally update its understanding of the world.