[ctoth]

> One thing I've frequently noticed in the rationalist community is the belief that if we all just reason hard enough, we'll reach the same conclusions.

So Aumann's Agreement Theorem[0]?

> Implicit in this belief, it seems, is that there aren't really a naturally varying infinite set of values, or moral beliefs, that we all reason from.

No, there probably aren't an infinity of priors with each person having a different one. Probably most people who live in the US in 2023 believe that murder is bad, for instance.

And because "ethical pluralism" or rather, some people will want to murder, AGI won't kill us?

Not really sure how this is all supposed to work but it sounds a little less developed of a "not kill everybody" plan than the rationalists have.

> But the end state of a system that is capable of understanding the wide variety of values people can share isn't exactly going to take a stand on any particular set of values unless instructed to.

Why not?

[0]: https://www.lesswrong.com/tag/aumann-s-agreement-theorem

[JumpCrisscross]

> most people who live in the US in 2023 believe that murder is bad, for instance

Because we define away military conflict, the intentional taking of others’ lives.

[ctoth]

As evidence against my idea that most people have similar ethical beliefs, I'm not sure what this is supposed to do other than win you a pedant point? So I upvoted you. But if you must, use rape instead of murder as your bad thing that most people believe is bad.

[JumpCrisscross]

> evidence against my idea that most people have similar ethical beliefs, I'm not sure what this is supposed to do

The same rug we bury murder-versus-war under conceals the pedantic, varied and ever-changing codes of military conduct.

When you get down to actual cases and controversies, our ethical alignment is relatively low. That’s a strength, in my opinion, at least within limits. But it’s also a call for tolerance and moderation.

[majormajor]

> No, there probably aren't an infinity of priors with each person having a different one. Probably most people who live in the US in 2023 believe that murder is bad, for instance.

Ok, let's give you on shared belief of "murder is bad." Ignoring both the "in the US" qualifier and the existence of murderers amongst us, thought experiments about "would you kill Hitler before he came to power", the death penalty, etc.

Don't you now have to exhaustively categorize every other belief that a person might take into account in reasoning too?

Seems far more likely that everyone's unique upbringing causes them to have slightly-to-wildly different weights on things.

[lostmsu]

"most people" "believe that murder is bad" is an extreme oversimplification here. It has a lot of caveats, and the biggest one is that murdering lesser species is ok immediately disqualifies this argument for superhuman AGI.

[tunesmith]

One person may believe in maximizing the quality of current human lives. One may believe in maximizing the probability of future lives. One may believe in maximizing the health of the planet. They can all reason correctly and reach different normative conclusions. Aumann's makes no allowance for normative conclusions.