[vintermann]

You're right that want is important. But what I think you miss (I may be wrong) is that "want" is not an empirical category.

Does a cloud want to rain? Or does it just rain because it obeys physical laws as part of a huge complex system?

Do I want to have a cup of coffee? Or a job? Or a girlfriend? Or am I likewise just doing things because my atoms obey physical laws as part of a huge complex system?

This is not a scientific question. It is a teleological question. That doesn't make it less important, on the contrary. It's the answers to teleological questions which make anything important, including the scientific questions which happen to be important.

So the question is not, "what can we do to make a machine want things", it's "When should we ascribe what a machine does to its own wants".

And my answer is "never". Not as long as it's a product of our wants, which they always will be.

[itworkslikethat]

Yes, you want to work, you want to have a cup of coffee. You may not like it, but you decided to do it and then executed a very complex series of steps to accomplish the goal. You're not a cloud, working/drinking is not an inevitable and direct physical process you're undergoing by yourself.

[vintermann]

I obviously agree that I have both will and wants.

But I can't say that the physical processes in me are in any fundamental way different, than physical processes anywhere else in nature. Most are deterministic, some may be inscrutable, and some may even be random or unknowable - but all those things can be said about the weather, too. You cannot derive my wants from my capabilities (unless you're prepared you ascribe wants to clouds as well, I guess).

[itworkslikethat]

The actual physical process of your body using the water once you ingest it is not much different, sure. But you can choose not to drink (and die some days later). A cloud can't choose not to rain.

[vintermann]

I'm pretty sure I can't will myself to thirst myself to death, actually.

But you're again missing the point. It feels to me like I have choices. I believe you're right, I have choices. You're kicking down an open door when you're arguing this.

What I'm telling you is that I didn't come to this belief empirically. And indeed I believe it is impossible to justify empirically.

[itworkslikethat]

You can. It takes some drugs, but you can. It's not uncommon for people to die because of dehydration at parties (taking stuff like extasis, etc). Regular users of methamphetamines also have this problem - they just don't realize they're completely dehydrated.

My point is that this comparison doesn't make sense. A cloud rains because it's undergoing a physical process. You drink because you decided to drink (and only then, inside your body, the water is undergoing a physical process). It's usually hard to decide not to drink, but still - there's a huge categorical difference in what's happening in either case.