| > but I find it fairly meaningless Why? I don't find your hint helpful; I have a pretty good idea what an epistemological state is but no idea what an "ethical state" or "aesthetical state" would be. The meaning seems perfectly clear to me: human nature, with all its cognitive biases and imperfections of memory and perception and limited thinking speed and imagination and so forth, makes it very unlikely that you will ever really be in the situation of having to choose between definitely-for-certain killing one person and definitely-for-certain letting one die, with definitely-for-certain no other options. Now, that isn't (as jerf already pointed out) EY's actual argument, it's a hypothetical argument he put in someone's mouth and described as a dodge. He is, I think, endorsing something along the same lines: It is unlikely that you will ever be in such a situation and, empirically, situations at all like that are very rare. So quite likely, even if you think you are in such a situation the probability that you actually are is low. On the other hand, you're extremely likely to encounter situations where you have the opportunity to harm people while convincing yourself you're doing good overall. Accordingly, it may very well be that net expected utility is optimized by having you follow principles like "the end justifies the means". Even when it seems to you that you're in an exceptional case where you shouldn't. In other words, consequentialists should sometimes behave like deontologists. But some hypothetical superintelligence (EY isn't AIUI talking about a perfected future hypothetical self, by the way) might well have a much better ability to tell what situation it's in and what options it has, and much less tendency to be corrupted and self-deceiving in the same ways as we are. If so, it would not be appropriate for it to operate on the principle that the ends don't justify the means -- at least, not if the ultimate goal is to maximize net expected utility. Consequentialist AI-designers might not do best to program their AIs to act like deontologists. I have no idea what makes you think that EY's aim is to say "look at me, I'm so clever"; for what it's worth, I find his argument clearer and less word-salad-y than yours. (Especially the weird paragraph about Jesus.) And no, your purported summary is not accurate, for at least the following reasons: (1) EY didn't say anything about hypothetical future versions of himself, and (2) as jerf pointed out EY said in so many words that the "I refuse to answer your question because I couldn't in that epistemological state" response is "a dodge" and that one ought to have better answers to such questions. (Though he does think -- as AIUI you do too -- that this specific question may not deserve a more serious answer.) |
If EY had meant: human nature, with all its cognitive biases and imperfections of memory and perception and limited thinking speed and imagination and so forth, makes it very unlikely that you will ever really be in the situation of having to choose between definitely-for-certain killing one person and definitely-for-certain letting one die, with definitely-for-certain no other options. then maybe he should have spelled all that out, don't you think?
I reassert that very little new was said in this article and what was said was wrapped in a ton of verbiage.
He says (and I paraphrase) - take this hypothetical utilitarian dilemma, then imagine this being that is qualitatively different from you or me. I imagine the being would respond thus owing to its special ability but as I am not worthy of a micron of its circuity I would have to choose otherwise as I do not have this special-ness. And he goes on to say, this so happens to turn out to coincide with the old maxim "the end doesn't justify the means" but I'm not saying that this is an intrinsic law or anything and I certainly wouldn't constrain our robotic overlords to it, they may very well judge it right to sacrifice one person now to save many later and I'd go along with that.
He might be saying: It is unlikely that you will ever be in such a situation and, empirically, situations at all like that are very rare. as you suggest but then again that does not seem to jibe with what he actually says: think the universe is sufficiently unkind that we can justly be forced to consider situations of this sort. and: But any human legal system does embody some answer to the question "How many innocent people can we put in jail to get the guilty ones?", even if the number isn't written down.
What is AIUI by the way? And I know I say future perfect hypothetical self at times and perfected other being at times but it doesn't alter what I'm saying - you'll grant that a superintelligence could theoretically maybe possibly fold all the remaining meat-machines into itself (don't you?), at least that appears to be one claim of singularity-types. Oh, I also find most of the singularity arguments compelling just in case you think I'm against super AIs or anything.