[skybrian]

Yes, more generally, I’m reminded of David Chapman’s essay, “No Cosmic Meaning” [1]. Thought experiments are a good way to depress yourself if you take them seriously.

But I think that utilitarianism has a vague but somewhat related problem in treating “utility” as a one-dimensional quantity that you can add up? There are times when adding things together and doing comparisons makes a kind of sense, but it’s an abstraction. Nothing says you ought to quantify and add things up in a particular way, and utilitarianism doesn’t provide a way of resolving disputes about quantifying and adding. Not that it really tries, because it’s furthermore a metaphor about doing math, which isn’t the same thing as doing math.

[1] https://meaningness.com/no-cosmic-meaning

[greiskul]

The big problem with utilitarinism, is that people think that a preference function for the utilitariam that is creating a given world is something simple. Then some people are like, no, it's more complex, we need to take into account X, Y and Z. But the truth is, no human being is capable of defining a good utility function, even for ourselves. We don't know all the parameters, and we don't know how to combine those parameters to add them up. So I would say that formal, proper utilitarinism, is not a metaphor for math: it is math. But is right now in the area of non constructive math.

Maybe our descedants will elevate it outside of that with computers someday. Cause the human brain with just pieces of papers and text, probably cannot do it.

[Gibbon1]

Also utilitarinism was created by people who were utterly unaware that the world is fundamentally chaotic. Instead they thought it could be represented by a system of linear equations.

It's fundamentally broken in practice.

[pdonis]

> utilitarianism has a vague but somewhat related problem in treating “utility” as a one-dimensional quantity that you can add up?

Yes, it does. This is one of the most common (and in my view, most compelling) criticisms of utilitarianism.

[at_a_remove]

One of the very muddled thoughts I have in my head, along with Goodhart's Law and AIs which blissfully attempt to convert the universe into paperclips, is that having a single function maximized as a goal seems to give rise to these bizarre scenarios if you begin to scan for their existence.

I have started to think that you need at least two functions, in tension, to help forestall this kind of runaway behavior.

[pdonis]

Even "two functions, in tension" still assumes that you can capture values as functions at all. But the reason ethics and morality are hard in the first place is that there are no such functions. We humans have multiple incommensurable, and sometimes incompatible, values that we can't capture with numbers. That means it's not even a matter of not being able to compute the "right" answer; it's that the very concept of there being a single "right" answer doesn't seem to work.

[at_a_remove]

I think that's what it will approach in the limit, yes, if you are talking about humans. For AIs, I think it will be somewhat less so, and that it would be preferable for the sake of predictability.