"Edit: In other words, how could it fail to grasp and obtain our very basic level of consciousness?"
"Argument from incredulity", among its many other flaws, has never in human history been a good guide to the future. Many, many things that people would find non-credible (since "incredible" has sort of wandered in meaning in the last century) have even so come to pass. "Look on my works, ye Mighty, and despair!"
It's phrased funnily, but it's not just an argument from incredulity.
I'm making a claim: that an entity with orders of magnitude more intelligence and knowledge would logically have a superset of our brain's functionality.
And that there's nothing magical about our brains. A super intelligence would crack the nut of human consciousness in its first attempt.
By what mechanism do you assume that an understanding of human consciousness is going to lead to anything like ethical behavior, given the enormous counterexamples given by our own behavior? Plenty of humans kill other humans fully aware that they are humans.
I assume that an understanding will lead to a capability, because that's the history of humanity's progress. First we understand a phenomenon, then we master it.
If humans ourselves understood human consciousness, we could probably replicate it in software today. This might end up being one of the ways a super intelligence comes into being.
"I assume that an understanding will lead to a capability, because that's the history of humanity's progress. First we understand a phenomenon, then we master it."
That's a category error, in that it doesn't even answer the question I asked. By what mechanism does an understanding of human consciousness by an AI necessarily and inevitably lead to moral behavior? As opposed to, for instance, using that understanding to accomplish its own immoral (or even merely non-moral) goals? Especially in light of the fact that it is very unlikely that you consider all humans to be moral, and that all such humans all provide existence proofs of intelligences that understand something, but for which that understanding did not necessarily and inevitably lead to moral behavior.
You appear to be proposing the certain (and rather ill-defined) existence of mechanisms that lead to morality that don't even work in humans.
On a similar note, by what mechanisms does an understanding of human consciousness by an AI necessarily and inevitably lead to moral behavior, when that AI is owned by an evil human? What mechanism, precisely, do you expect to save us when President Trump Jr., or Clinton II, or whatever other human you currently believe to be extremely evil, orders the AI to work out the most effective plan to exterminate whoever they consider their enemies this week? Which, since I asked you to fill in whom you consider evil, includes you in the target list. What, exactly, is going to save you in that case? By what mechanism is the AI going to go "Oh, no, not staunch, I can't kill staunch, that would be immoral."
If you don't have an answer to that... and you don't... some caution may be warranted in AI research.
I recommend you read this novel "Blindsight" it's about aliens, but the author is a biologist with a focus on neurology and the bibliography reads like a PhD thesis.
TLDR: consciousness might be training wheels for intelligence and not of terminal value.
Intelligence and ethical behavior are orthogonal. You can expect a super-intelligence to develop a super-understanding of ethics. That does not imply ethical behavior.
Why do you think they're orthogonal though? Empirically, it seems like smarter humans are more ethical. Same seems true of high intelligence vs low intelligence animals too.
Orthogonal is exaggerating, I'll agree that the angle may not be exactly 90 degrees, but it's nearer that than 0.
>Empirically, it seems like smarter humans are more ethical.
"Seems" deserves some emphasis. Who is more likely to wind up in handcuffs, a smart thief, or a dumb one?
>Same seems true of high intelligence vs low intelligence animals too.
It's not exactly clear what constitutes ethics in animals. Applying typical human ethics; Chimpanzees have murdered their social rivals, and dolphins sometimes enjoy tormenting other animals.
What makes you think smarter humans are more ethical? It certainly not always the case. There are smart sociopaths and criminals. Smart people made a world with thousands of nukes in it.
That would be for you to demonstrate, but even if we accept it to be true for the sake of a discussion, it is not relevant, because there is no reason to believe a non-human intelligence would choose to accept those ethical percepts that humans would prefer.
Unless humanity's level of ethics derive from our level of intelligence. In which case, our only complain would be that we're too stupid to understand its superior ethics.
The same way a dog might not understand the ethics of a doctor injecting life-saving medicine into it. The doctor is acting super ethically but cannot explain it to the dog.
We are under no logical obligation to accept that the ethics of entities that can outsmart us are superior, even if your premise is correct - and as that premise itself is merely speculative at this point, it would be wise to be prudent.
Edit: In other words, how could it fail to grasp and obtain our very basic level of consciousness?