[derefr]

A thought: if someone offered me a trillion dollars to kill a person, I would get trillion-dollar loan, invent brain-uploading and emulation technology with it, copy the person into the machine, and then kill them.

To put that more simply: past a certain point, the idea of marginal value in a cost/benefit analysis breaks down, because the benefit "changes the game." (That is, creates a discontinuity in the valuation curve.) A million dollars "changes the game" of your life. $100 doesn't.

[cduan]

It's never clear to me why people answer thought experiments like this in this manner. The point of the question "would you kill someone for a trillion dollars" is to tease out an answer to an underlying philosophical question, such as, "is the act of killing as we know it a moral absolute wrong, or is it merely an act with great negative utility that can be compensated by sufficient positive utility?"

Yes, there are always creative ways to avoid really answering the question (such as doing a brain upload, as suggested), but these answers do nothing to answer the underlying question, which is what the thought experiment is really trying to get at.

[derefr]

And I've always ignored that sort of question—there's no such thing as morals or utility, really, only what animals are programmed to think in their animal brains. Robotic arms don't care when other robotic arms die—and neither do we. Paperclips don't care about increasing in number, even if someone is trying to make it so. What we have to ask is not "what does the universe say about right and wrong?" (because the answer is "nothing") but rather "what do our minds say about right and wrong?"—that is, "what is the human utility function?"

The problem with that, of course, is that even within our species, we have many different (and mutually-exclusive!) utility functions; sociopaths, for example, calculate theirs noticeably differently. So, it still ends up turned into a problem of cultural meta-ethics. That is, it's no longer a matter of "who do we shun and revile?" but "how do we get along?" or perhaps "do we want to get along?" (Which brings me to this: http://lesswrong.com/lw/y4/three_worlds_collide_08/)

[cduan]

That's not really a reason to avoid answering the question either. Regardless of the mechanism by which your mind arrives at the answer (be it animal instincts, brain chemistry, whatever), you are indeed capable of coming up with an answer, and even if you believe the answer says nothing about the universe in general, it still says something about you and your own mind. And I'm fairly interested in knowing what your mind--and the minds of other people--have to say about these sorts of questions.

To be a little more clear, your answer (yes or no) is of little value to me. What I really want to know is why you answer yes or no--is it because of some general principle you're applying, because of a gut instinct, because someone told you to say that, or something else?

EDIT: Based on the reply below, I'm not being clear. There are lots of reasons I'd like to know your reasoning process, beyond trying to generalize it to people in general. Among other things, I'd like to know whether I should be worried about going to dinner with you (particularly if your answer is "Sure, I'd kill for even just a dollar").

[derefr]

But that's a question for neuropsychology, isn't it? The question of what is good for an individual person won't be solved by conversation and introspection, because people are, on average, very bad at understanding themselves and the reasons for doing the things they do (they indeed have reasons to give, but these reasons rarely stand up as falsifiable hypotheses for predicting future actions, so they're mere rationalizations and should be discarded.)

Instead, the question of what an individual cares about will be solved by coming up with a technique to look at a person's brain and tell them, definitively, what values they care about at that moment in time. Anything said about individual ethical belief until then is just sophistry.

[ntoshev]

The answer may come from other places than neuropsychology. We as carriers of values live and evolve in an ecosystem where the values of those around us and how they fit with ours matters. Maybe we can derive some evolutionary stable strategies representing moral laws from a swarm of agents constantly playing Prisoner's dillema and games with other rewards against each other.

[derefr]

But the mechanisms by which we play those games will be evident in the brain, and the games themselves will be evident in our expectation-program memory. (I'm assuming here that sufficiently-advanced neuropsychology will be able to analyze the "software" running on each brain, not just the firmware, but even if that field ends up with a different name, that's what I'm talking about here.)

We may be able to model some sort of "objectively-good" cooperative-evolutionary game-players using mathematics, but those models would not necessarily represent us; there's nothing that says we're even evolutionarily stable as a species over the long term ;)

[Dunearhp]

Why does someone need a reason not to answer a silly question.

[henrikschroder]

there's no such thing as morals or utility, really

Just to counterpoint the frenzy of moral nihilism and relativism I suspect will appear, I respectfully disagree, I'm a moral universalist, i.e. morals do exist, moral judgements can be true or false.

But I also realize that this is a question of philosophical faith, and arguing about that is usually pretty futile. :)

[derefr]

A few questions:

1. Given the laws of physics, could you derive your morality? 2. If we didn't exist, and in our place were an alien species that, say, ate their babies[1], would that make the universe contain less utility as a whole?

[1] The same story I linked to above: http://lesswrong.com/lw/y4/three_worlds_collide_08/. Who is in the right? Do we have the "moral imperative" to destroy the baby-eaters? Do the third species have a moral imperative to destroy us?

[henrikschroder]

1) No, the only way to get my morality is by asking me. As for whether or not a specific moral judgement is true or not, no, you can't derive that from the laws of physics either. What an incredibly boring universe that would be if that was the case.

2) You're asking me to sum up and compare the values of two complete civilizations? That's a pretty tall order. Also, in one of the alternatives I wouldn't exist, that's a lot less utility right there.

[derefr]

You just said you were a moral universalist—that goes against the concept of there being a "your" morality. Either morals are universal (in which case, you don't need a specific living being to ask in order to figure them out), or they're, at least in some part, subjective. Further, you seem to be explicitly arguing against yourself with the "in one of the alternatives I wouldn't exist" statement—what system of universal morals would privilege your existence over the existence of someone who is better in every way, except for not being you?

What I said above was that "morals or utility" are subjective to an individual—which agrees with your point. I said "and thus don't really exist" because the definitions for epistemic morality or universal utilitarianism require them to be universal—and they're not, so they don't exist as defined. (And a system that accepts subjective morality is usually just called "ethics", by the way.)

[btmorex]

For an individual you can make those proclamations, but individuals don't operate in a vacuum. There is such a thing as morals and utility in a society. There is value in participating in a society. Therefor, there is value is having individual morals and calculating the utility of actions at least on a societal level.

[derefr]

Right—that's, effectively, the difference between "cultural meta-ethics" and "moral relativism." What I'm saying is that, on an individual level, ethics is a problem that will be completely solved by neuropsychology: once we find out our particular utility function, we just have obey it optimally. Thus, Ethics as a field of endeavor should drop that kind of individual-level moral quandary, and focus on inter-societal quandaries, since that's what we'll really need to figure out—how to handle, and interact with, societies that have different utility functions than our own (including the ones we end up creating ourselves using genetic engineering, AI, and so forth.)

[billswift]

From the most recent post on LW http://lesswrong.com/lw/2aa/virtue_ethics_for_consequentiali...

Humans are not inherently expected utility maximizers, they're bounded agents with little capacity for reflection. Utility functions are great and all, but in the words of Zack M. Davis, "Humans don't have utility functions."

[jimbokun]

"once we find out our particular utility function, we just have obey it optimally."

Is this function computable?

[nzmsv]

But the morals of a society are not constant. They are a function of the morals of the individuals living in a society. What was perfectly acceptable in the 12th century is barbaric now. In a century or two we will seem barbarians to our descendants. (And they will still be saying that their generation lost all morals and humans will soon die out.)

[bruceboughton]

I wonder for what reasons our society will be judged barbaric by our descendants.

[Shorel]

Irak, the SUV vehicles, the bullying in schools, people parasiting the welfare system, the predatory banking practices, all those things seem barbaric to me.

[olliesaunders]

I came to the same conclusion a while ago. I like to phrase it as: morality is a function of evolution.

[JoeAltmaier]

My old minister said it well: "We have all the morals we can afford"

[xtho]

The example in the article is a good example how such thought experiments are sometimes turned into jokes for bored misanthrope/misogynist/latent homosexual men. At first, the situation could be interpreted as an exchange of a years of happy life for sleeping with somebody she considers attractive and would probably sleep with anyway. This problem then is turned into what would be her price if she slept with men for a living.

If this is your understanding of philosophy, I'd say you're misusing philosophy for dealing with personal problems.

[btmorex]

If you're going to invent scenarios, you should invent realistic one's. For example, I'm fairly confident that you couldn't solve the problem of brain duplication for a trillion dollars. The NIH budget in 2010 was ~32 billion which isn't close to a trillion, but it's a lot closer to a trillion than we are to duplicating a brain.

On a philosophical level though, a lot of people would probably disagree with your proposed actions. For me, personally, I wouldn't want to "influence" (vaguely kill, but also any of these "duplication" scenarios) someone else's mortality for any amount of money.

[chancho]

You could cure a lot of cancer for $1 trillion. By turning the offer down, are you sure you're not "influencing" someone else's mortality? (That's a rhetorical question, of course. I don't have an answer.)

[btmorex]

So, I did say "any amount of money", not any amount of lives. I do think it's morally okay to trade one life for many (definitely ambiguous in terms of amounts because I don't feel like trying to figure that one out). In fact, I don't think you could justify war without also justifying this moral decision so I don't think it's very rare even if people don't often think about it.

[jimbokun]

"I do think it's morally okay to trade one life for many"

So you're a Spock, not a Kirk.

[JoeAltmaier]

CLoning would be cheaper/possible

[MichaelSalib]

You couldn't replicate memories and consciousness though. And cloning involves a long lead time since we only know how to clone embryos. So from your family and friends' perspective, it would be like you died, since they're not going to enjoy the company of your clone for a few decades and even then it won't have your memory or much of your personality.

[philwelch]

So for a trillion dollars, you wouldn't kill someone, only restrict them to living inside a stationary computer for eternity without their consent.

[derefr]

The important word in my analogy was "invent." Given that the technology exists, lots of other things would happen—an emulation could be plugged into arbitrary "sensory" inputs, so a simulated universe could be created for it; that simulated universe would seem like an inviting prospect for those that do not wish to continue a life that mostly consists of sitting around on the Internet anyway, so more people (first-adopters) would upload as well, even if their "real" bodies continued separately; the large number of ems would provoke discussions of em rights, which would lead to internationally-funded medical programs to create two-way bridges between our reality and the simulations... and so on.

That's what I mean by "discontinuity": you can't give a cost-benefit analysis to something like a trillion dollars, because one trillion-dollar investment can completely alter the course of civilization with its knock-on effects.

[philwelch]

You're still seriously inconveniencing some guy, and doing something highly significant to him without his consent.

But it was mostly a quibble :) I agree with your central argument.

[ojbyrne]

Given sufficient technology, what exactly is the difference?

[btmorex]

Well, you're making decisions about someone else's mortality. Maybe they wouldn't want to live forever with everyone else that they knew dying? Maybe they would want to have a physical existence in the form of a human body. You're denying them that.

There are a ton of similar questions.

[derefr]

I think the parent's phrase "sufficient technology" was meant to be interpreted as more than my "brain uploading and emulation"—i.e. an infinite Matrix landscape, synchronized with a doorway in the real world, that would gradually quantum-de-/re-materialize them as they progressed further into or out of it, so that, on the outside of the door, they were real people made of real matter, but ten or fifteen feet inside they were completely virtual. Assuming a wide-enough door, you could bring in your house, your car, whatever else you'd like, and bring them out again if you wanted to live a mortal life. (But, if you got your leg blown off in reality, you could always go in and, sufficiently digitized, apply a patch from a record of your previous body and walk back out, good as new.)

The problem with speculative fiction is that it needs a technological conflict. Worthwhile utopias don't do that.

[lazugod]

The rest of the word might hesitate to consider the person "not dead" when their body, their instantly recognizable human feature, has been taken away from them.

[dkersten]

Would that amount vary much if the person you were to kill was someone you cared about?

[Whippet]

That gets to the crux of the matter doesn't it.

If your morals are based on principle, then no matter who the person was, your decision would remain the same.

I believe/hope that I am the type of person who would not take anothers' life no matter what the circumstances because I want to be a person of principle. If there were some situation where I did take a life, then I am not the principled person I think I am.

Take the example of Google and their motto,'Don't be Evil'. It was a nice bit of marketing, but when the value of doing business with a repressive government was high enough, they and other company's apparently had no problems working with said governments to continue the repression of it's citizens.

Now, you may say, Google has left that country. But, it was not due to the moral repugnance they felt about repressing the citizens, but the fact they were under attack by agents/citizens of said government and felt it was no longer in their interest 'financially' to stay.

[nicolas_t]

A bit out of topic, I think on the question of China many people have a black and white view of the situation...

In the case of google, they came to China but actually provided a less censored service than baidu (by writing in the search pages that some results were left out because of censoring thus attracting attention to it)... So in what way was their behavior evil?

Additionnaly, they never started any blogging service in China that would have put them in a situation where they had to give information to the government about political activists (unlike yahoo who gave such information)...

So, I don't think Google did anything evil in that case...

[btmorex]

They traded freedom of speech for ~1.3 billion people for a decidedly limited amount of money. To me that's a moral calculation and they chose money.

[nicolas_t]

In what way did they trade freedom of speech? Did people in China have more freedom of speech before they came to China? Could they have forced the chinese government to limit censoring?

Of course not... The only thing they could do and did is give another reliable search engine that censored less than their competitors...

[JoeAltmaier]

No, I understand it required collusion with China to make it impossible to resolve Google.com to a different search engine. If they did nothing, folks could Google with the same engine the rest of us do. Try to remember, your ancestors may have fought and died to stay free. Google certainly did not extend themselves in any way to promote this ideal. This discussion here is probably not available in China.

[sailormoon]

That is utter hyperbole. Google never had the "freedom of speech for ~1.3 billion people" to trade. They do not and probably never will have anything like that kind of power. I am far from a Google fanboy but if you're going to criticise them, please keep it reasonable.

[btmorex]

That is reasonable. I'm simply asking them to not do business with a repressive government. Maybe, you don't think freedom of speech is important, but I do.

As for what kind of effect they could have: Google is an internationally recognized brand that is most certainly known in China (even without being the dominant search engine). Refusing to do business with China would be hard for the state-run media to explain away.

Of course, I think a lot of people really, really want to believe that Google is different from every single other multinational organization that has ever existed. It's not.

[btmorex]

Well said. I always thought it was odd that people cheered Google when they pulled out of China even though they only did that because the Chinese government was hacking them (most notably not becase the Chinese government doesn't care about freedom of speech).

[sailormoon]

the Chinese government doesn't care about freedom of speech

Oh, I think the CCP cares very deeply about freedom of speech. Just maybe not in the way that you'd like ; )

[jedbrown]

Terrible example, but the point that cost-benefit analysis is nonlinear is indeed valid. Suppose that instead of being offered money, you were threatened with losing it. This change is immaterial for small amounts, but highly relevant for large amounts (e.g. as the amount approaches your net worth/lifetime earning potential).