[JoshTriplett]

Because if we build it correctly, it won't be a person at all, and won't have a value function orthogonal to our own. Building something that can self-improve and solve problems doesn't mean we need to build something that has independent will, motivation, or values. And it's a lot safer not to.

[StavrosK]

Sure, but "an AI that's smarter than humans" implies that. I'm sure we'll build many algorithms that are better than us at specific tasks, and we already have.

[JoshTriplett]

Even building algorithms that are better than us at any general-purpose task still doesn't require giving those algorithms agency or independent values of their own.