[Padding]

> Historically, slavery never worked out for anyone, at least not in the long term.

This is a rather contentious point I think. "We" wouldn't be the "developed" part of the world had we not enslaved the rest of it.

> And the idea that we can even in principle enslave potentially god-like intelligences seems ultimately futile

You're assuming AIs will have a will of their own to begin with, which might not be necessarily true. You can't enslave something that has no preference towards whichever course reality takes.

Or to put it another way, since you're talking about god-likeness, what difference does it make to you wether you're figuring out the optimal way to route traffic through town or correcting the trajectories of missiles en-route to kill millions? It's not like you'd have anything "better" to do, given that you're all knowing, and it's not like either of those alternatives will have a significant impact on the universe in the long run anyways.

[Udo]

> You can't enslave something that has no preference towards whichever course reality takes.

I'm adamant that this is not true. For example, you can drug a person with something that makes them not care about the course of reality, and which makes them compliant with pretty much anything. Doing that is still abuse, it's not suddenly OK because the drugs caused the person not to care. I would even argue it's an especially egregious form of abuse.

> what difference does it make to you wether you're figuring out the optimal way to route traffic through town or correcting the trajectories of missiles en-route to kill millions

You could apply the same argument to humans, and indeed in many situations we do consider these two to be equivalent - for example while doing service in the military.

> It's not like you'd have anything "better" to do, given that you're all knowing

There's a difference between an intelligent individual and a mindless calculator, and that difference is the ability to reflect on your own existence and the existence of others. Humans are a good example, because while we're capable of mindless indifference, we also have the capability to reflect and be ethical. It's culture that makes the difference here. I'm advocating we'll bring AGI up in a culture conducive to ethics.

[Padding]

> Doing that is still abuse

Based on what?

I'm not for drugging people and using them as zombies. And obviously drugging people against their will is already forcing onto them something they don't want. But, striping away all human context such as dignity, culture, etc., and assuming an absence of opposition from the person/entity in question, I can't think of any actual reason why it would indeed be "wrong" to drug someone/something into happiness.

There is no absolute measure for happiness, and many already seem to give up some of their freedom in the range feelings and desires they experience in order to feel happier (with anti-depressiva) or be more successuful (with things like Ritalin).

Taking someone utterly unhappy with their life and putting them in some matrix-like environment where they can both experience joy and still be useful is, I think, an alright thing to do.

> the ability to reflect on your own existence and the existence of others

We exist.

You can't make any stronger claim than that without involving some form of (arbitrary) value/belief system.

Why would an all-knowing entity bother with having a set of beliefs it values, if there's no formal reason/need for them?

> I'm advocating we'll bring AGI up in a culture

Culture is a slippery slope.

There are so many different ones of them which conflict, creating the potential for conflict and retaliation. Which is where we imperfect humans are at.

But culture is also arbitrary (as far as I can tell). Why would some all-knowing entity prefer one culture over another? And if indeed it did, what would hat say about those other cultures? Would genocide all of a sudden be aceptable? Slippery slope..

[Udo]

> I can't think of any actual reason why it would indeed be "wrong" to drug someone/something into happiness

You switched out the original premise on me there ;)

> You can't make any stronger claim than that without involving some form of (arbitrary) value/belief system.

You can end any discussion by invoking this principle. This is the essence of the incompleteness theorem applied to every day reasoning. Somewhere at the bottom of every perspective, there are some arbitrary axioms. It's a way of saying "that's just your opinion, man" and you'd be right of course.

> Why would some all-knowing entity prefer one culture over another?

Why would a human? I suspect the word culture might have different connotations for either of us.

[Padding]

> Why would a human?

Because evolution graced (cursed?) us with a reward system and parents that utilize (abuse?) it.

Having something capable of high-level reasoning, while free from the desires, fears, moods and other emotions humans suffer is part of the reason why we're looking into AI right?

> that's just your opinion, man

Maybe? I have 5 fingers on my hand - is that an opinion? Maybe it is, because what's an opionion anyways? But who would dispute it?

> Somewhere at the bottom of every perspective, there are some arbitrary axioms

Well not quite. "Arbitrary" perhaps in a formal sense, since logic doesn't care about specific universes but truths that hold in all of them. Yes, you still end up having to settle for implicit definitions somewhere along the line (what a finger is, what method you use to count them, etc.). But there nevertheless is some difference between merely assuming something exists, and assuming what should exist.

Something that is all-knowing would be able to figure out the difference between premises that indeed need to be true for our universe to exist (like me needng having 5 fingers right now), and those that humanity merely believes or wants to be true (like it being good that I not use those fingers to poke out someones eyes).