[igravious]

If epistemological state means anything at all it means (the possibility of) being able to hold a certain belief or acquire certain knowledge. It is as meaningful or as meaningless as using the phrase "ethical state" or the phrase "aesthetical state" and I stand by that claim.

If EY had meant: human nature, with all its cognitive biases and imperfections of memory and perception and limited thinking speed and imagination and so forth, makes it very unlikely that you will ever really be in the situation of having to choose between definitely-for-certain killing one person and definitely-for-certain letting one die, with definitely-for-certain no other options. then maybe he should have spelled all that out, don't you think?

I reassert that very little new was said in this article and what was said was wrapped in a ton of verbiage.

He says (and I paraphrase) - take this hypothetical utilitarian dilemma, then imagine this being that is qualitatively different from you or me. I imagine the being would respond thus owing to its special ability but as I am not worthy of a micron of its circuity I would have to choose otherwise as I do not have this special-ness. And he goes on to say, this so happens to turn out to coincide with the old maxim "the end doesn't justify the means" but I'm not saying that this is an intrinsic law or anything and I certainly wouldn't constrain our robotic overlords to it, they may very well judge it right to sacrifice one person now to save many later and I'd go along with that.

He might be saying: It is unlikely that you will ever be in such a situation and, empirically, situations at all like that are very rare. as you suggest but then again that does not seem to jibe with what he actually says: think the universe is sufficiently unkind that we can justly be forced to consider situations of this sort. and: But any human legal system does embody some answer to the question "How many innocent people can we put in jail to get the guilty ones?", even if the number isn't written down.

What is AIUI by the way? And I know I say future perfect hypothetical self at times and perfected other being at times but it doesn't alter what I'm saying - you'll grant that a superintelligence could theoretically maybe possibly fold all the remaining meat-machines into itself (don't you?), at least that appears to be one claim of singularity-types. Oh, I also find most of the singularity arguments compelling just in case you think I'm against super AIs or anything.

[gjm11]

AIUI = "as I understand it". Sorry for any confusion.

> If epistemological state means anything at all ...

It is clear (to me, anyway) that by "epistemological state" Yudkowsky means "state of beliefs and knowledge" rather than what you say is the only thing it can possibly mean. Why do you think the only thing it could mean is what you state?

(I think he should have said "epistemic" rather than "epistemological".)

> If EY had meant ... then maybe he should have spelled all that out

Maybe. But what he wrote was pretty long already, and "since I am running on corrupted hardware" (which is what EY did write) amounts to much the same thing. There's nothing a writer can do to guarantee that every single reader will understand correctly.

> I reassert that very little new was said in this article

So you do. But you're reading only a portion of it; you make claims about its overall purpose which are clearly contradicted by the article itself (hint 1: "to me this seems like a dodge"; hint 2: "I now move on to my main point", followed by a statement of that point which is not anything like "how can I best respond to trolley problems?" or "our robotic overlords will be vastly superior to ourselves"); you ignore large parts of it altogether. Why should anyone care whether, treating it thus, you find anything new in it?

> and what was said was wrapped in a ton of verbiage

Well, yes, Yudkowsky is not the most concise writer in the world. I think that may be partly because he's found that being terser gets him misinterpreted more often. From your consistently inaccurate paraphrases and summaries here, it seems to me that his main problem probably wasn't excess verbosity.

> that does not seem to jibe with what he actually says: ...

Situations where you're in the sort of epistemic position described in trolley problems are very rare. Situations where you can, and maybe should, harm some people to benefit others are not so rare.

I dare say there are ways in which a superintelligence could "fold all the remaining meat-machines into itself". It's not so clear that any of them would result in there being a superintelligence which is a "version of" any of those meat-machines.

I neither know nor care exactly what your attitude to super AIs is. I do think, for what it's worth, that pretty much everything you've said here on the subject has an unpleasantly sneering tone which you might want to lose if you don't want to give the impression of being "against super AIs or something".

[igravious]

Hi gjm11,

I want to impress upon you sincerely that I am not sneering. I do, as I have said (and I stick by it), dislike EY's writing on a stylistic level. I'm not going to hide that. This is partly down to personal taste. I think that style says a hell of a lot about the substance of a person's thought†. Having read countless academic and academic-type essays where people try to mask their lack of knowledge with a hailstorm of jargon and moreover lack the decency to take the time to go through their reasoning in plain and simple language. It really bugs me, EY may not be guilty of it but it sure looks like it to me.

The very first commenter to his essay said swap 'perfect-tech-being' for 'angel' and you get a philosophical debate as old as utilitarianism itself but you're right it is a nice singularity slant on an old problem, and in this way it is novel but we could go through the whole of philosophy of mind, or even philosophy of religion and substitute daemons and angels for super-ais and the claims already made therein would not change much.

Both you and jerf have taken the time to show me the thinking behind the surface and I thank you for that. I will be more generous in future.

Epistemological properly means 'of or relating to a theory of knowledge' so "epistemological state" cannot really mean "state of beliefs and knowledge", as you say "epistemic state" would have been a lot closer to this. I think it is best to avoid words like epistemological unless you happen to be Per Martin-Löf‡ or someone of that calibre because we all fail at wielding such terms judiciously (including me of course) and that's not meant to be snarky!.

Concision! Oh yes please. What are the virtues of philosophical writing? Brevity. Clarity. Humour. A sharp use of metaphor. A Himalayan perspective :)

†http://ebooks.adelaide.edu.au/s/schopenhauer/arthur/lit/chap...

‡check this out for a jaw-dropping walk through the gardens of logic / epistemology http://docenti.lett.unisi.it/files/4/1/1/6/martinlof4.pdf

[gjm11]

There's one significant difference between posing the problem in terms of AIs and posing it in terms of angels: if AIs are going to exist we'll have to design them[1] and whoever's designing them will be trying to ensure that their behaviour fits (something like) our values; whereas if angels exist, they were designed by someone else whose values may be quite different from our own and it's no business of ours to decide how they should behave.

[1] Perhaps indirectly.

Yeah, I like concise writing too. Concise and clear is even better. Concise, clear and funny, better still. Yudkowsky doesn't do too well on conciseness, but I think he does just fine on clarity and humour. (You might want to bear in mind that the article linked from here is part of a lengthy ongoing series (perhaps I should say: series of series) that EY was writing at the time; it's doubtless clearer when read in the context of the rest of it.)

Although "epistemic" would have been better, I really don't think "epistemological" need have been such a roadblock. If someone refers to an organism's "biological makeup" or "physiological condition", I hope it would be clear that they mean the kind of makeup/condition with which biology/physiology is concerned, rather than the organism's pet theories about biology and physiology. So also with "epistemological state".

What do other users of such terminology mean by it? I just asked Google for <<<"epistemological state">>> and of the first page results I reckon: first one is this discussion; second one is EY's meaning; third is ambiguous; fourth is yours; fifth is EY's; sixth is EY's (and says in so many words: 'Philosophers tend to suppose that one's "epistemological state" is constituted by beliefs'; the authors are philosophers); seventh is yours; eighth is ambiguous but I think nearer EY's; ninth is a sort of hybrid, nearer to yours; for the tenth (of which I can see only the snippet Google provides, the rest being behind a paywall) I can't tell. Some of those hits are from people whose use of philosophical language I wouldn't trust for an instant, but at least four seem reputable. (I am not sure whether to be relieved or alarmed that the ones that look reputable to me on other grounds are also the ones that favour EY's usage; perhaps I'm suffering from some bias or something.) It seems like EY's usage is pretty reasonable. I still think "epistemic" is better; as you may have noticed, he's now changed it.

[Eliezer]

> (I think he should have said "epistemic" rather than "epistemological".)

Fixed.