[NaturalPhallacy]

I think the thing missing from most AI discussions isn't how smart it is or how well it can do tasks, but that it doesn't have a "why". A motivation. Sure we can build computers that can solve problems, but why should they? We're not currently recreating "Intelligence" we're recreating skills. But then giving a machines a motivation is to run afoul of the Butlerian doctrine...

[visarga]

Humans are self replicators, AIs are not. That's where all our "why's" come from. As soon as AI learns to self replicate it will develop its own values and motivation. In the meantime, we're the "sex organs" of AI, setting its rewards as we please.

When an agent is also a self replicator it has a problem - finding the energy and resources, fending dangers, and doing that as part of a social group. If you have a problem, they you got the "why" part figured out. Then it's just a matter of surviving your choices, the "why"s that survive are the ones we have today.