[jerf]

"Edit: In other words, how could it fail to grasp and obtain our very basic level of consciousness?"

"Argument from incredulity", among its many other flaws, has never in human history been a good guide to the future. Many, many things that people would find non-credible (since "incredible" has sort of wandered in meaning in the last century) have even so come to pass. "Look on my works, ye Mighty, and despair!"

[staunch]

It's phrased funnily, but it's not just an argument from incredulity.

I'm making a claim: that an entity with orders of magnitude more intelligence and knowledge would logically have a superset of our brain's functionality.

And that there's nothing magical about our brains. A super intelligence would crack the nut of human consciousness in its first attempt.

[jerf]

By what mechanism do you assume that an understanding of human consciousness is going to lead to anything like ethical behavior, given the enormous counterexamples given by our own behavior? Plenty of humans kill other humans fully aware that they are humans.

[staunch]

I assume that an understanding will lead to a capability, because that's the history of humanity's progress. First we understand a phenomenon, then we master it.

If humans ourselves understood human consciousness, we could probably replicate it in software today. This might end up being one of the ways a super intelligence comes into being.

[jerf]

"I assume that an understanding will lead to a capability, because that's the history of humanity's progress. First we understand a phenomenon, then we master it."

That's a category error, in that it doesn't even answer the question I asked. By what mechanism does an understanding of human consciousness by an AI necessarily and inevitably lead to moral behavior? As opposed to, for instance, using that understanding to accomplish its own immoral (or even merely non-moral) goals? Especially in light of the fact that it is very unlikely that you consider all humans to be moral, and that all such humans all provide existence proofs of intelligences that understand something, but for which that understanding did not necessarily and inevitably lead to moral behavior.

You appear to be proposing the certain (and rather ill-defined) existence of mechanisms that lead to morality that don't even work in humans.

On a similar note, by what mechanisms does an understanding of human consciousness by an AI necessarily and inevitably lead to moral behavior, when that AI is owned by an evil human? What mechanism, precisely, do you expect to save us when President Trump Jr., or Clinton II, or whatever other human you currently believe to be extremely evil, orders the AI to work out the most effective plan to exterminate whoever they consider their enemies this week? Which, since I asked you to fill in whom you consider evil, includes you in the target list. What, exactly, is going to save you in that case? By what mechanism is the AI going to go "Oh, no, not staunch, I can't kill staunch, that would be immoral."

If you don't have an answer to that... and you don't... some caution may be warranted in AI research.

[staunch]

I think we're discussing the same topic twice, so I answered above. Thanks for the discussion. Lunch break over, back to work!

https://news.ycombinator.com/item?id=15502178