[Analemma_]

This is pretty much word-for-word the reasoning behind Roko’s basilisk, which made its proponents an internet laughingstock for a decade, but is a surprisingly tricky thing to actually refute if you accept the premise that AGI is in fact coming.

[AstroBen]

It seems quite easy to refute—why would it punish anyone?

We would pose 0 threat at that point to any super intelligence, and I highly doubt it would have anything like a human grudge. It's just a case of anthropomorphizing it

[ToValueFunfetti]

The premise is that it's trying to influence human behavior before it becomes powerful by punishing them afterwards. Like how part of the reason you give the guy a ticket is to substantiate the disincentive for the speeding he already did. It's not an emotional thing.

What's sketchy is that you and it come to this arrangement without communicating. Because you are confident this thing that has total power over you will come into existence and will have wanted something of you now, you're meant to have entered a contract. This is suspect and I think falls prey to the critiques of Pascal's wager- there are infinite things superintelligence might want. But it's certainly tricky.

[ares623]

It’s just another form of god to believers. And what is a god if not a tool for punishment