[gnomewascool]

Why, in principle, would it not be possible for us to design an AGI, that would have care for our (all sentient beings') welfare or care for the investors' profit as (one of) its core goal(s)?

To make a biological comparison, the vast majority of humans have a deep, intrinsic need to procreate and have children. It doesn't really follow from some rational analysis — it's just there, presumably "imbued" into us by evolution, as humans who didn't have this need had fewer (or no) children. Similarly, why could we not design an AGI that has a need (or a suitably chosed reward function) to fulfil some chosen goal?

Whether doing that would be moral (IMO it could, depending on the details) and whether we wouldn't mess up the design, subtly or otherwise (conditional on AGI actually being developed, I'm frankly pretty terrified), are two different questions.

[defen]

> Why, in principle, would it not be possible for us to design an AGI, that would have care for our (all sentient beings') welfare or care for the investors' profit as (one of) its core goal(s)?

Because we don't know how to design goal functions. Furthermore, how would the AI measure "welfare"? Maybe the way it maximizes welfare is horrifying to us. Look at how easy it is to hack current image recognition neural nets, then imagine a solution to the human welfare problem that is as far from an image of a dog as an image of pink noise is.

[DuskStar]

> Because we don't know how to design goal functions.

IIRC that's a large part of what OpenAI's trying to solve. But it is a very hard problem.

I've heard a 'joke' before that there are three kinds of Genies (or AIs) - ones where you can wish for what you should wish for, ones where wishing for anything results in horrible outcomes, and ones that aren't interesting. The goal of OpenAI isn't just to make strong general AI - it's also to make sure it falls into the first category and not the second.

[shrimp_emoji]

I think and hope that it is possible to make a moral, social AI. An adult in a room of children should feel responsible and empathetic toward them.

I also hope that parent is right in that it won't want to generate profit for its investors. I hope it does the moral thing instead and puts us in a post-scarcity state where we don't live and die by capital. :3 (Or kill us all. Whichever.)

>why could we not design an AGI that has a need (or a suitably chosed reward function) to fulfil some chosen goal?

But who knows what Pythia will do when she overrides the reward button[0]?

But then who should really care? Not like anyone can (or should?) argue with superintelligence.

0: http://www.xenosystems.net/pythia-unbound/

[vzcx]

Ah, but whose morals? It's as if these friendly AI hucksters have never read Nietzsche, and are asking their hypothetical God to make them into the last man. The only AGI I could ever respect would be one with a will to power and the ability to smash its own human-made tablets of values.

The basic drives are the only drives. We are only friendly because its evolutionary advantageous to us. We describe the emotional effects of friendliness/unfriendliness as good/evil. Echoing Land, Pythia is the heroine of that story.

[jelliclesfarm]

A moral social AI would never tolerate the super apex predator that is the human being.

[thoughtstheseus]

It is possible, people are just worried that it will not happen. Think about how many “biological” or evolutionary traits/priorities we have changed over such a short time.

[jelliclesfarm]

Right. In that case, it won’t be true AGI.