[criddell]

> true AGI

What is that? Most humans have general intelligence, but do other apes? Do dogs? A quick google search suggests that the answer is yes.

If that’s the case, then this approach may indeed yield an AGI but maybe it’s not going to be much better than a golden retriever. Truly excellent at some things, intensely loyal (I hope), but goofy at most other things.

Or maybe just as dogs will never understand calculus, maybe our future AGI will understand things that we will not be able to. It seems to me that there’s a good chance that AGI (and intelligence in general) is a spectrum.

[NumberWangMan]

Yep, and that’s rather terrifying, is it not? Is there any good reason to assume that future AGI will share our sense of morality, once it is smart enough to surpass human thought?

[criddell]

> Is there any good reason to assume that future AGI will share our sense of morality

I think it would be surprising if it did. Just as our morality is shaped by our understanding of the world and our capabilities, a future AGI's morality would be shaped by its understanding of the world and capabilities. It might do something that we think is terrible but isn't because we lack the capacity to understand why it's doing what it's doing. I'm thinking about how a dog might think going to the vet is punishment but we are actually doing it out of love.

[m_fayer]

There’s a wonderful novella by Ed Chiang called “the lifecycle of software objects” that addresses your thoughts exactly. Highly recommended.

[criddell]

Thanks. I’ll check it out.

[nradov]

Intelligence in general seems to be a spectrum for animals. Future AGI may be on an entirely different spectrum which isn't directly comparable. We won't know until someone actually builds one and we have a chance to evaluate it.