[jerf]

"As long as this super intelligence leads to super ethics and super consciousness"

That is begging the question, in the original sense. The entire question here is how do we do that when there is no a priori reason to assume that a superintelligence will have either of those things, and indeed, most if not all of our best techniques today would certainly not produce those things if they did lead to explosive intelligence.

[staunch]

1. Is super goodness an inevitable result of super intelligence?

2. Is super intelligence an inevitable result of creating self-evolving software?

My feeling is that the answer is yes to both, but I agree that these are not settled questions.

[eli_gottlieb]

>1. Is super goodness an inevitable result of super intelligence?

No. Evidence: $PERSON_YOU_HATE is very intelligent. Human moral instincts/reasoning may be neuroscientifically simple, having some core mechanism that "unfolds" across sensorimotor datasets. This is very plausible, because we can already see core sensorimotor systems that include interoception, and a core affective system to move the sensorimotor systems along trajectories designated valuable by the interoceptive circuits. These are core mechanisms that operate across hugely hierarchical models that capture datasets across multiple scales of space, time, and variation.

However, none of that is any reason to think our particular combination of affective, interoceptive, and sensorimotor machinery - especially our brain's "bias" towards "mirroring" and other hyper-social reasoning - will be universal to all possible brains.

This especially applies to disembodied "brains" like "artificial intelligences", which, not having a bag of meat to move around, won't have the same kind of reward and interoceptive processing as us at all. This means they won't have anything remotely like our emotional makeup, which means that even with careful reinforcement training of social reasoning, they will not have humanoid motivations, by default.

[staunch]

> Evidence: $PERSON_YOU_HATE is very intelligent.

In every case I can think of, I think the person's lack of intelligence is the reason I dislike them. That they do not understand my ethical problem with them is the problem. Or their calculus is off, which also seems to be a lack of intelligence from my POV.

> ...they will not have humanoid motivations...

That's probably a good thing. We should want them to be better than us, which is probably necessarily unlike us in many ways. The important thing is that they're ethical, even if we're incapable of comprehending or recognizing it.

[eli_gottlieb]

>The important thing is that they're ethical

And, non-tautologically, where do you think the ethical knowledge and motivation come from?

[jerf]

The problem is that all the evidence we have basically points to no for 1, and we have a big lack of evidence for 2. Self-evolving software is a thing that exists and I studied over a decade ago, which was already decades old, and it has not resulted in super intelligence.

[staunch]

What kind of evidence is there against #1?

For #2, we're at the very beginning of computing, so calling any technology impossible this early makes no sense.

[jerf]

"What kind of evidence is there against #1?"

All intelligence produced to date has had nothing like "morality" from anything I've seen, not even the basic seeds of it we can see in simple animals in the wild. There's even been some hand-wringing articles about it about AI-based approaches locking minorities out of loans and such, for instance. Certainly the military has not reported any problems to date with their AI research declining to kill people because they have moral problems with it, nor have any of the self-driving car teams reported that their job has been eased by the fact that the self-driving car AIs have spontaneously generated a sense of morality that makes them strive to not hit people.

Heck, the very idea that this would happen sounds downright silly when I say it.

I still say you're basically arguing from incredulity. You can't imagine an intelligence that isn't intrinsically human, therefore they can not exist. Plenty of the rest of us can imagine intelligences that aren't human. I say there's even some we live with, such as bureaucracies, that are super-intelligences composed of humans that still manage to have inhuman behaviors and pathologies; how much moreso an intelligence composed not of humans.

[staunch]

Chimps hunt down and kill monkeys for food. It's really hard for humans to watch but no chimp would ever even consider the ethics of this, even though they do have a level of ethics (e.g. fairness among themselves)

We're more intelligent than chimps, so we understand that monkeys are pretty evolved and suffer a lot when you eat them alive. Chimps aren't kept up at night by the screams of their victims.

A healthy human that tortures other intelligent lifeforms will often suffer severe mental anguish as a result. Only a human with a severe mental disorder can torture other lifeforms without remorse.

So what does that tell us about intelligence? That an intelligent understanding of a monkey's suffering leads to more ethical behavior.

But humans aren't that intelligent. We still let other people do unethical things on our behalf, like raising animals in terrible conditions for meat. The more intelligent (and knowledgeable) a human being, the more likely they are to have a problem with the suffering of factory farmed animals.

[eli_gottlieb]

>So what does that tell us about intelligence? That your understanding of a monkey's suffering leads you to more ethical behavior.

Rubbish. We don't empathize because we're intelligent. We empathize because we're empathetic. That's a specific mental capacity, separate from causal inference.

[Zaak]

Have you heard the paperclip maximizer thought experiment? Here is a good video on the subject: https://www.youtube.com/watch?v=tcdVC4e6EV4

[staunch]

This is the risk that a narrow intelligent machine will go nuts. A super intelligence would know not to do stupid stuff, by definition.

[Zaak]

The problem isn't about doing stupid stuff. The problem is about doing very smart stuff when your core values massively conflict with humanity's continued existence.