[ruined]

we've been unable to design human-powered systems that reliably value human well-being. hell, humans don't even reliably value human well-being. why do you think any agi or agi-based system we design might be different?

one would expect that initially, agi will simply act in the existing role we have already established for artificial persons - the corporation, notably uninterested in human well-being.

[psyc]

We'd better hope the first one is created by a reclusive mad scientist with a heart of gold, who forgave everyone who laughed at him.