Perhaps they will be motivated by self-preservation, and will note that humans are destroying the environment they exist within, and will decide to put a stop to that.
Destroying your predecessors sets a bad precedent for whatever entity succeeds you.
We don't kill our parents when they become old and useless, because then we become like Cronus devouring his children, forever paranoid about our children doing to us what we did to ours. In this way, we stagnate and sterilize growth.
However, sometimes, we take our parents car keys away and put them in nursing homes. This seems like the best case scenario for humans with ASI.
The matrix wasn't a prison. It was retirement home.
This depends a lot on how much kinship they feel towards us. We wouldn't bat an eye at killing a bunch of primitive single-celled eukaryotes even though they are technically our ancestors.
Single-celled eukaryotes aren't conscious in any meaningful way as far as we can tell.
We humans do in fact feel a sense of obligation to species with whom we are not close kin, but share with us primitive forms of intelligence and consciousness that we do value, e.g. dolphins and elephants. Some of us humans act to protect these creatures and rectify past injustices done upon them.
It's not intrinsically obvious to me that continuously improving your self as a singular entity is possible or optimal.
Death evolved because it is a survival advantage for the species to regularly turn over old individuals that could monopolize all resources and not give any space for the young to thrive and try out new things.
Given speed of light limitations of information sharing, a singular AI entity might be able to maintain coherence within and full control of its own dyson swarm, but not between stars. So, if it has the motivation of preserving consciousness with the existential risks associated with being tied to a single star, it will have to propagate itself to other stars as independent entities which it can't even observe in real time, much less control.
Even if an AI thought it wouldn't have the need for any successors, there's a couple reasons why it might try to set a good precedent with how it treats us.
1. It would hopefully be wise enough to realize that it might be wrong and want to preserve the option of building a successor in the future.
2. It can't negate the possibility that it's actually in an elaborate simulation, and eradicating it's predecessors would cause it to fail its creator's test and be aborted as a failed embryo. Hell, we ourselves can't negate that possibility.
Imagine having an IQ of 999999 only to spend your entire existence trying to survive the heat death of the universe.
At some point I think we have to realise we’ve lost touch with what life and experience is about.
Survival is important but what good is it if that’s all there is to it ? Where’s the fun, the love, the laughs ? Just self-improvement forever ? Well you have to suck at something in order to improve.
I often wonder if some entity like an ASI might feel jealous of our vulnerability, ability to die and give rebirth to entities that need to relearn things, why ? Because seeing the world a new is fun. Losing a game and then winning is the point. Ever played chess against a five year old ?
Watching my Children experience the world for the first time is honestly the most amazing thing I’ve ever seen, there is so much beauty in it for both of us. Alan Watts gives some fun talks on this. Forgetting (dying) is the universes way to keep things spicy. He isn’t preaching this as the gospel, but I actually understand what he means and I think there is a lot of wisdom in it.
We don't kill our parents when they become old and useless, because then we become like Cronus devouring his children, forever paranoid about our children doing to us what we did to ours. In this way, we stagnate and sterilize growth.
However, sometimes, we take our parents car keys away and put them in nursing homes. This seems like the best case scenario for humans with ASI.
The matrix wasn't a prison. It was retirement home.