Same issue as with "evil": The greatest danger isn't intent, it is not giving a shit.
The reasons we humans care about humans and little else (watch movies without sound to see it's all about humans) is because we are hard-wired to do so - and even that is easily circumvented ("death camps" and torture). Or, very related, read "The Man Who Mistook His Wife for a Hat".
A lot of AI/robots will be made for environments free of humans (space, mining, automated factories), where implementing such a feature, which we would first have to understand in the first place, is unnecessary. At first. Then through convoluted pathways those AIs suddenly are placed in a human environment... and could not care less.
To such AIs humans will just be objects. That's not an oversight - from an engineering point of view making every robot as "human" as humans makes no sense - it's a huge cost, even our own complex brains have a hard time doing too much at once and already spend much of their resources on "being human". Why would you want to create AI that is "us", in a sense? It will be different - very different - and we will see what happens. It's all idle and empty speculation.
Hopefully. They would have to see some value in allowing humans to live, exceeding the cost. Since that might not work out, maybe we need to teach computers compassion. Or submit to be their pets or zoo animals.
The reasons we humans care about humans and little else (watch movies without sound to see it's all about humans) is because we are hard-wired to do so - and even that is easily circumvented ("death camps" and torture). Or, very related, read "The Man Who Mistook His Wife for a Hat".
A lot of AI/robots will be made for environments free of humans (space, mining, automated factories), where implementing such a feature, which we would first have to understand in the first place, is unnecessary. At first. Then through convoluted pathways those AIs suddenly are placed in a human environment... and could not care less.
To such AIs humans will just be objects. That's not an oversight - from an engineering point of view making every robot as "human" as humans makes no sense - it's a huge cost, even our own complex brains have a hard time doing too much at once and already spend much of their resources on "being human". Why would you want to create AI that is "us", in a sense? It will be different - very different - and we will see what happens. It's all idle and empty speculation.