Hacker News new | ask | show | jobs
by wapapaloobop 3655 days ago
The problem with your scenario is that it equates humans with animals, but the difference that matters is between people (humans, AGIs) and non-people (primitive replicators, animals).

We don't commit murder to get at rocks despite the fact that weren't designed with 'consideration for human values'. We learn our values, just as an AGI would, not through programming or lack thereof, but through parenting, education and culture. These latter things aren't optional since minds can't exist without them.
1 comments
outworlder 3655 days ago
> We learn our values, just as an AGI would

Yes, but there's a problem. An AGI would be a completely alien mind. It does not have to have the same values(or any values we could relate to), just like you wouldn't expect a super intelligent spider to have any morality, no matter how good the parenting was.

link
wapapaloobop 3655 days ago
An AGI would have to begin with the same values because without them he couldn't learn anything. Our values draw on the surrounding culture and an individual that grows up outside of any culture (e.g. a feral child) isn't capable of functioning. There may be different cultures but there is no alien culture. Thus there could be no alien mind.
link