[Aurornis]

The AI 2040 writing and the AI 2027 before it both sprang from within the rationalist communities. Scott Alexander (of the popular rationalist Slate Star Codex / Astral Codex Ten blog) played a role in both articles, though he says he chose to leave his name off of this one.

AGI and AI doom scenarios have been evergreen topics in rationalism for years. Long before we had enough compute power to do anything at all useful with generative AI. Many of the rationalist scenarios about future AIs had very religious themes where the AI became so powerful that it was effectively a god.

Rationalists had concept's like Roko's Basilisk, a thought experiment where a future all-powerful AI might choose to punish anyone who predicted the arrival of AI but failed to contribute to the arrival of AI.

The rationalist communities took this thought experiment so seriously that the biggest rationalist forum of the time formally banned discussion of Roko's Basilisk for 5 years. They believed it was an "info hazard" because once you knew about this risk, you either had to contribute to bringing about this future AI god or you risked being punished by it in the future.

If this is new to you, you might think I'm exaggerating or making things up to make a group look bad, but this was such a core belief that it has its own Wikipedia page: https://en.wikipedia.org/wiki/Roko%27s_basilisk

The people writing AI 2027 and AI 2040 were immersed in these cultures. That's why they think that AI is the primary topic that anyone is talking about in elections, just like the Bitcoin people all thought crypto was the only relevant election topic.

The GP comment is getting attacked, but there really is a heavy religious fervor background to the AI fears in the rationalist community. The religious angle and extremism of the forums is downplayed when in crosses over into mainstream topics, but it's been there from the start.

[pnt12]

First I was intrigues by the rationalist movement, but I was deeply disappointed. To put it bluntly, seems to me like geeks LARPing as philosophers, and not with rational argument as they promise, but rationalizing their sci fi fanfictions.

[layla5alive]

Yes, calling themselves "rationalists" seems to be the biggest tell. That they wish they were that. Yet their behavior reflects anything but.

[ThrowawayR2]

If I recall correctly, the text of Yudkowsky's response to Roko's Basilisk used to be part of the Wikipedia page but was removed and not even a reference link was provided even though the response was mentioned in the body of the article. For posterity's sake, from https://rationalwiki.org/wiki/Roko%27s_basilisk/Original_pos... here it is in all its hilariously unhinged glory:

"One might think that the possibility of CEV punishing people couldn't possibly be taken seriously enough by anyone to actually motivate them. But in fact one person at SIAI was severely worried by this, to the point of having terrible nightmares, though ve wishes to remain anonymous. I don't usually talk like this, but I'm going to make an exception for this case.

Listen to me very closely, you idiot.

YOU DO NOT THINK IN SUFFICIENT DETAIL ABOUT SUPERINTELLIGENCES CONSIDERING WHETHER OR NOT TO BLACKMAIL YOU. THAT IS THE ONLY POSSIBLE THING WHICH GIVES THEM A MOTIVE TO FOLLOW THROUGH ON THE BLACKMAIL.

There's an obvious equilibrium to this problem where you engage in all positive acausal trades and ignore all attempts at acausal blackmail. Until we have a better worked-out version of TDT and we can prove that formally, it should just be OBVIOUS that you DO NOT THINK ABOUT DISTANT BLACKMAILERS in SUFFICIENT DETAIL that they have a motive to ACTUALLY BLACKMAIL YOU.

If there is any part of this acausal trade that is positive-sum and actually worth doing, that is exactly the sort of thing you leave up to an FAI. We probably also have the FAI take actions that cancel out the impact of anyone motivated by true rather than imagined blackmail, so as to obliterate the motive of any superintelligences to engage in blackmail.

Meanwhile I'm banning this post so that it doesn't (a) give people horrible nightmares and (b) give distant superintelligences a motive to follow through on blackmail against people dumb enough to think about them in sufficient detail, though, thankfully, I doubt anyone dumb enough to do this knows the sufficient detail. (I'm not sure I know the sufficient detail.)

You have to be really clever to come up with a genuinely dangerous thought. I am disheartened that people can be clever enough to do that and not clever enough to do the obvious thing and KEEP THEIR IDIOT MOUTHS SHUT about it, because it is much more important to sound intelligent when talking to your friends. This post was STUPID.

(For those who have no idea why I'm using capital letters for something that just sounds like a random crazy idea, and worry that it means I'm as crazy as Roko, the gist of it was that he just did something that potentially gives superintelligences an increased motive to do extremely evil things in an attempt to blackmail us. It is the sort of thing you want to be EXTREMELY CONSERVATIVE about NOT DOING.)"