[igravious]

So let me get this straight.

The author is saying that his answer to the the classic hypothetical dilemma in ethics of "is it right to do harm (murder one) to prevent a greater harm (death of five)" which usually framed in utilitarian-ish debates is:

a hypothetical incorruptible super awesome version of me would murder the innocent person but as I am not that type of being (and only that type of being can answer the question) I am not going to answer the question

But why not go all in and pose it thus: "is it better for one person to suffer eternal suffering to free all others from any type of suffering"? And that's why Jesus died for our sins you know. And that really happened. And Jesus was smarter than a hypothetical incorruptible super awesome version of you. Therefore ... I'm not sure where this is going.

Maybe what I'm trying to say is (and by all means argue the toss with me and don't shoot me down) Eliezer Yudkowsky sounds a lot smarter than he actually is.

[jerf]

That is not an adequate summary of the point. The point that you are addressing is that if we are to truly take this as a question about "what should a real person do?", the question can be rephrased without loss as "You are a person standing before the track and you know with 100% certainty that if you flip the switch that one person is 100% likely to die and if you do nothing that five people are 100% likely to die." and his response is that it isn't even possible in theory for a person to know these things with 100% certainty. The key phrase is "I can't occupy the epistemological state you want me to imagine".

I would also draw your attention to the first sentence of the next paragraph: "Now, to me this seems like a dodge." This isn't the core point of the essay, and the more I stare at it, the more it does seem like it's five paragraphs accidentally ripped from another essay ("And now the philosopher comes" -> "Now to me this seems like a dodge"); if you just cut those five out entirely it seems more focused, and those five paragraphs can spin off into another interesting essay. (One that would, I think, conclude that this is actually just a way of rephrasing the idea that philosophical hypotheticals are actually useless by virtue of being impossibly overspecified which itself comes from impossible oversimplification, and in general the hypothetical question "What if an absolutely mathematically impossible thing happened?" is not a fruitful line of thought.)

[igravious]

Great response.

I did not highlight the distinction that in this class of hypotheticals that the lesser harm requires action and that the greater harm requires one to simply do nothing, to stand idly by as it were - oh my god, I can here the voice of my prof in my head from days gone by as I clarify this point. Still, the causal link remains, one can either choose to act or not (or insist that you cannot even begin to play the game as was done here). But apart from that action/inaction subtlety I have to disagree with you here, it is an accurate if not entirely straight-faced summary.

Look, you might have a good working understanding of "can't occupy the epistemological state" (oh really, why? because I haven't achieved the level of perfection of my future hypothetical self) but I find it fairly meaningless. Hint: substitue epistemological with ethical or even aesthetical to see if such an assertion becomes any more meaningful. Note: I am not saying that I am positioning myself against the "you can't even begin to play (or, I'm not playing) the game" stance or some variation thereof as my response to this dilemma would be probably something along these lines given my aversion to hypothetical thought experiments such as this which I feel contribute very little to the debates in morality and ethics.

This isn't the core point of the essay. But this is not the case surely. The essay makes many points, sure, but this chain of reasoning is I believe fairly central and although it could be excised I believe that the author formulated the whole essay this way for a reason. This post-singularity being's properties are analysed in the light of a very classic problem in philosophy. If you look at the comments you will see that a poster points out that "regular" philosophers invoke mythical beings such as 'angels' or 'ideally rational agents' which are non-tech versions of what is going on here. I don't think it's a dodge, it doesn't even seem like a dodge and the author didn't even need to point this out. Where I'm coming from is that this ground has been covered and it has been covered in language that is not obfuscated. The jargon salad does nothing more than communicate "look at me, I'm so clever" which is why I claim that the author sounds smarter than he actually is.

just a way of rephrasing the idea that philosophical hypotheticals are actually useless by virtue of being impossibly overspecified which itself comes from impossible oversimplification This would be something a logical positivist would say. It's something I'm very inclined towards. I agree that hypotheticals like this generate a good amount of noise and heat but they fail to be constructive or advance our understanding of ethical questions beyond perhaps showing what ethical norms a person subscribes to, to whit: all life is sacred and one is commanded by a supreme being to do no harm, all life has intrinsic worth/value so you shall never through action do harm, you shall optimize for the greater good, and so on and so on.

[gjm11]

> but I find it fairly meaningless

Why? I don't find your hint helpful; I have a pretty good idea what an epistemological state is but no idea what an "ethical state" or "aesthetical state" would be.

The meaning seems perfectly clear to me: human nature, with all its cognitive biases and imperfections of memory and perception and limited thinking speed and imagination and so forth, makes it very unlikely that you will ever really be in the situation of having to choose between definitely-for-certain killing one person and definitely-for-certain letting one die, with definitely-for-certain no other options.

Now, that isn't (as jerf already pointed out) EY's actual argument, it's a hypothetical argument he put in someone's mouth and described as a dodge. He is, I think, endorsing something along the same lines:

It is unlikely that you will ever be in such a situation and, empirically, situations at all like that are very rare. So quite likely, even if you think you are in such a situation the probability that you actually are is low. On the other hand, you're extremely likely to encounter situations where you have the opportunity to harm people while convincing yourself you're doing good overall.

Accordingly, it may very well be that net expected utility is optimized by having you follow principles like "the end justifies the means". Even when it seems to you that you're in an exceptional case where you shouldn't. In other words, consequentialists should sometimes behave like deontologists.

But some hypothetical superintelligence (EY isn't AIUI talking about a perfected future hypothetical self, by the way) might well have a much better ability to tell what situation it's in and what options it has, and much less tendency to be corrupted and self-deceiving in the same ways as we are. If so, it would not be appropriate for it to operate on the principle that the ends don't justify the means -- at least, not if the ultimate goal is to maximize net expected utility. Consequentialist AI-designers might not do best to program their AIs to act like deontologists.

I have no idea what makes you think that EY's aim is to say "look at me, I'm so clever"; for what it's worth, I find his argument clearer and less word-salad-y than yours. (Especially the weird paragraph about Jesus.)

And no, your purported summary is not accurate, for at least the following reasons: (1) EY didn't say anything about hypothetical future versions of himself, and (2) as jerf pointed out EY said in so many words that the "I refuse to answer your question because I couldn't in that epistemological state" response is "a dodge" and that one ought to have better answers to such questions. (Though he does think -- as AIUI you do too -- that this specific question may not deserve a more serious answer.)

[igravious]

If epistemological state means anything at all it means (the possibility of) being able to hold a certain belief or acquire certain knowledge. It is as meaningful or as meaningless as using the phrase "ethical state" or the phrase "aesthetical state" and I stand by that claim.

If EY had meant: human nature, with all its cognitive biases and imperfections of memory and perception and limited thinking speed and imagination and so forth, makes it very unlikely that you will ever really be in the situation of having to choose between definitely-for-certain killing one person and definitely-for-certain letting one die, with definitely-for-certain no other options. then maybe he should have spelled all that out, don't you think?

I reassert that very little new was said in this article and what was said was wrapped in a ton of verbiage.

He says (and I paraphrase) - take this hypothetical utilitarian dilemma, then imagine this being that is qualitatively different from you or me. I imagine the being would respond thus owing to its special ability but as I am not worthy of a micron of its circuity I would have to choose otherwise as I do not have this special-ness. And he goes on to say, this so happens to turn out to coincide with the old maxim "the end doesn't justify the means" but I'm not saying that this is an intrinsic law or anything and I certainly wouldn't constrain our robotic overlords to it, they may very well judge it right to sacrifice one person now to save many later and I'd go along with that.

He might be saying: It is unlikely that you will ever be in such a situation and, empirically, situations at all like that are very rare. as you suggest but then again that does not seem to jibe with what he actually says: think the universe is sufficiently unkind that we can justly be forced to consider situations of this sort. and: But any human legal system does embody some answer to the question "How many innocent people can we put in jail to get the guilty ones?", even if the number isn't written down.

What is AIUI by the way? And I know I say future perfect hypothetical self at times and perfected other being at times but it doesn't alter what I'm saying - you'll grant that a superintelligence could theoretically maybe possibly fold all the remaining meat-machines into itself (don't you?), at least that appears to be one claim of singularity-types. Oh, I also find most of the singularity arguments compelling just in case you think I'm against super AIs or anything.

[gjm11]

AIUI = "as I understand it". Sorry for any confusion.

> If epistemological state means anything at all ...

It is clear (to me, anyway) that by "epistemological state" Yudkowsky means "state of beliefs and knowledge" rather than what you say is the only thing it can possibly mean. Why do you think the only thing it could mean is what you state?

(I think he should have said "epistemic" rather than "epistemological".)

> If EY had meant ... then maybe he should have spelled all that out

Maybe. But what he wrote was pretty long already, and "since I am running on corrupted hardware" (which is what EY did write) amounts to much the same thing. There's nothing a writer can do to guarantee that every single reader will understand correctly.

> I reassert that very little new was said in this article

So you do. But you're reading only a portion of it; you make claims about its overall purpose which are clearly contradicted by the article itself (hint 1: "to me this seems like a dodge"; hint 2: "I now move on to my main point", followed by a statement of that point which is not anything like "how can I best respond to trolley problems?" or "our robotic overlords will be vastly superior to ourselves"); you ignore large parts of it altogether. Why should anyone care whether, treating it thus, you find anything new in it?

> and what was said was wrapped in a ton of verbiage

Well, yes, Yudkowsky is not the most concise writer in the world. I think that may be partly because he's found that being terser gets him misinterpreted more often. From your consistently inaccurate paraphrases and summaries here, it seems to me that his main problem probably wasn't excess verbosity.

> that does not seem to jibe with what he actually says: ...

Situations where you're in the sort of epistemic position described in trolley problems are very rare. Situations where you can, and maybe should, harm some people to benefit others are not so rare.

I dare say there are ways in which a superintelligence could "fold all the remaining meat-machines into itself". It's not so clear that any of them would result in there being a superintelligence which is a "version of" any of those meat-machines.

I neither know nor care exactly what your attitude to super AIs is. I do think, for what it's worth, that pretty much everything you've said here on the subject has an unpleasantly sneering tone which you might want to lose if you don't want to give the impression of being "against super AIs or something".

[igravious]

Hi gjm11,

I want to impress upon you sincerely that I am not sneering. I do, as I have said (and I stick by it), dislike EY's writing on a stylistic level. I'm not going to hide that. This is partly down to personal taste. I think that style says a hell of a lot about the substance of a person's thought†. Having read countless academic and academic-type essays where people try to mask their lack of knowledge with a hailstorm of jargon and moreover lack the decency to take the time to go through their reasoning in plain and simple language. It really bugs me, EY may not be guilty of it but it sure looks like it to me.

The very first commenter to his essay said swap 'perfect-tech-being' for 'angel' and you get a philosophical debate as old as utilitarianism itself but you're right it is a nice singularity slant on an old problem, and in this way it is novel but we could go through the whole of philosophy of mind, or even philosophy of religion and substitute daemons and angels for super-ais and the claims already made therein would not change much.

Both you and jerf have taken the time to show me the thinking behind the surface and I thank you for that. I will be more generous in future.

Epistemological properly means 'of or relating to a theory of knowledge' so "epistemological state" cannot really mean "state of beliefs and knowledge", as you say "epistemic state" would have been a lot closer to this. I think it is best to avoid words like epistemological unless you happen to be Per Martin-Löf‡ or someone of that calibre because we all fail at wielding such terms judiciously (including me of course) and that's not meant to be snarky!.

Concision! Oh yes please. What are the virtues of philosophical writing? Brevity. Clarity. Humour. A sharp use of metaphor. A Himalayan perspective :)

†http://ebooks.adelaide.edu.au/s/schopenhauer/arthur/lit/chap...

‡check this out for a jaw-dropping walk through the gardens of logic / epistemology http://docenti.lett.unisi.it/files/4/1/1/6/martinlof4.pdf

[Eliezer]

> (I think he should have said "epistemic" rather than "epistemological".)

Fixed.

[jerf]

The debate has gone on great without me, but the only thing I would point out and verify is that in the context of Elizier's writing, as was pointed out, the "epistemological state" is definitely going to be the assigned probabilities the entity is carrying around internally for Bayesian updates. He may not have spelled that out this time but where the term might be fuzzy in other people's hands I do feel like I know fairly precisely what he means, and where the fuzziness may be isn't relevant to today's debate. (Also, I'm just alluding to the fact that I think I know what it means, this is not itself an explanation, just a labeling.)

[igravious]

Heh, welcome back :)

Okay - then I need to brush up on this Bayesian stuff, care to point me in the right direction? I only have undergrad level maths only so nothing to tricky please !! I really haven't meant to raise anybody's hackles here.

[jerf]

http://yudkowsky.net/rational/bayes

[JoachimSchipper]

Eliezer's answer seems to be "do no harm, even when this seems to be beneficial, as your judgment is likely to be wrong".

For instance, quite a few dictators seize power and suppress dissent because they honestly believe that's the best for the people. It's all too easy to convince oneself that something that is convenient for oneself but harmful to another is, ultimately, "for the greater good".

Of course, exceptions to the "do no harm" rule do exist. However, the probability of the current case being an exception may be vanishingly small, even if you are convinced it is an exception. If this probability is indeed very small, the "do no harm" rule produces a better expected outcome than a "pragmatic/utilitarian" point of view for a human (imperfect) agent.

In the philosopher's "100% sure" case, a true mathematical weighting can be made and (true) utilitarism wins; but no human can ever be that sure.

[secret]

There's a difference between sacrificing another and sacrificing yourself.

[igravious]

May we never be placed in that position ever in our lives.

[sp332]

Did you miss the bit where Jesus asserted power over death and came back to life after three days? The sacrifice was a blood sacrifice, not a soul sacrifice. Eternal suffering doesn't come into it.

[yters]

Theologically Jesus is to have suffered all that humans can suffer. So, even though He resurrected in 3 days, that doesn't mean He couldn't have experienced an eternity of torment. By His own account, He experienced complete abandonment by God.

[sp332]

It said he was tested (tempted) in all ways. Hell isn't a test or temptation, I don't think it counts.

[igravious]

Haha. No, I didn't miss that bit. I had it pretty well drummed into me as a kid in good old catholic Ireland. I'm no theologian but I was told he died for our sins which was awfully nice of him. I'm sure everything was roses post-mortem.

I was merely bringing the thought experiment to its logical utilitarian conclusion and showing that in doing so it has some mythical/biblical/primordial resonances.