I think we will soon find that what really makes us "human" is the shared chemo-biological composition of our selves with the rest of the ecosystem and evolutionary hierarchy.
When you cuddle a puppy you can actually smell the infant hormones on them... why... because we share evolutionary biology. How do you teach a computer to do that?
You can think of the body as "data" and the brain as nothing more than a database that let's you query it. AI might give a better database... but until it gets the same chemo-informatic data... well good luck.
Good point about the necessity of embodiment. I would go one step further and consider the environment, which is the source of evolution and knowledge, a simple environment like Atari games can't even begin to compare with the human society and world we experience. What makes us human has a lot to do with the dynamics of interacting with the other humans, an AI would need to be part of society to experience that.
When you cuddle a puppy you can actually smell the infant hormones on them... why... because we share evolutionary biology. How do you teach a computer to do that?
You can think of the body as "data" and the brain as nothing more than a database that let's you query it. AI might give a better database... but until it gets the same chemo-informatic data... well good luck.