[saulrh]

> Prime Intellect

Ah, yes, Prime Intellect, the AGI that went foom and genocided the universe because it was commanded to preserve human civilization without regard for human values. A strong contender for the least evil hostile superintelligence in fiction. What a wonderful thing to name your AI startup after. What's next, creating the Torment Nexus?

(my position on the book as a whole is more complex, but... really? Really?)

[robertclaus]

You may as well just go with Roko's Basilisk.

[cmrx64]

Least evil… strong words.

[saulrh]

It did host a successful and substantially-satisfying human civilization, at least until it let a couple of presumptuous self-important anarchoprimitivists kill it and genocide its subjects. Even if it was only a temporary and unstable illusion of alignment, that's one more values-satisfying civilization than the overwhelming majority of paperclippers manage. So yeah. Good? No. Least evil? Maybe.

[rep_lodsb]

>until it let a couple of presumptuous self-important anarchoprimitivists kill it and genocide its subjects

That could have just been their private simulation. As far as I remember, it wouldn't even have outright lied to them, just let them believe they talked it into destroying itself.

[gryfft]

GP did specify least evil hostile SI.

[QuesnayJr]

After reading that Torment Nexus post you didn't have the urge to name an AI product Torment Nexus? Really?