[AgentME]

Anything with much greater capabilities than us that would find our resources useful and that doesn't explicitly have our best values at heart would be a danger to us. European settlers were not good news for the Native Americans. Early humans were not good news for many great apes.

(It might be tempting to imagine this from a higher-level perspective and assert that each of these successors were somehow more interesting or had more fulfilling lives than what they replaced, and so therefore these successions might be ultimately good in the long run, but that's not meant to be part of this analogy. We won't have some genetic lineage with AGI; it will probably be more alien to our mind's design than actual aliens produced by evolution like us would be. The AGI might not build a world replacing ours where many of its kind have lives we would find fulfilling or interesting. It might care just about making the world completely predictable and safe, empty of outside threats and life, and hibernating in a place of safety. Plenty of animals live solitarily and might do this if they were naively uplifted into superintelligence. Humans and social animals instinctually believe in the goodness of friendship and society because evolution saw we were good at surviving in groups and molded our instincts to encourage us to do that. AGIs won't necessarily have that if we don't design that into them (or set up the process that designs them to do that).)