[JoshTriplett]

> I think Transcendence did a halfway decent job of showing some possible applications of friendly AI.

Interesting, I hadn't seen that one. While that doesn't give a very good impression of what the timescale could be (though it actually seems fairly plausible for an uploaded human that just runs faster with more data but doesn't attempt iterative self-improvement), it does suggest at least some possibilities in ways I haven't seen in other movies, and in relatively few short stories.

> That said, I don't think I'd constitute the conflation of any human mind and supercomputing as friendly.

"Friendly AI" is a specific term of art, referring to AI that values sentient beings, and whose value function involves modeling and satisfying the value functions of sentient beings. (Its opposite, "unfriendly AI", refers to any AI that doesn't value sentient beings; that doesn't just mean "actively hostile", but also simply "doesn't explicitly care", either one of which would create an existential threat. For instance, a paperclip-building AI that sees people as another source of matter that isn't paperclips yet.)

Almost all sentient beings tend to hold their own continued existence in high value.

[pnwhyc]

My concerns in regard to fusing human and machine aren't necessarily because I question the supportive nature of humanity. It's more so that there is no telling what a brain would do with that much power. It's not implausible to say that such a being might disassociate with the human race altogether. Furthermore, innate urges such as survival and production of offspring would still be strongly intact, probably much more prevalent than any idealistic thoughts floating around in the frontal lobe, hence posing a threat of the classic "robot takeover."

Comparatively, teaching/programming a system to have morality seems like a safer bet than giving the reins to a being whose psyche was sculpted by evolution.

[JoshTriplett]

I'd certainly agree that we wouldn't want the the most powerful system in the world to have originated as an uploaded human with all their cognitive biases and undocumented value functions, as opposed to an intentionally designed system.