[throw2500]

Yudkowskyist rationalism is a prime example that what you simply banish out of hand may percolate back up through your ideas -- unless you know enough about whatever it is you're trying to banish to recognize it.

The thing they tried to banish is religion: it simply morphed into another form. "Those who do not learn about religion are doomed to repeat it", as it were.

[c1ccccc1]

I'm always mildly surprised when I encounter people on the internet talking about Roko's basilisk as something that Eliezer Yudkowsky / "the rationalists" believe in. Very few people ever took it seriously, and Yudkowsky wasn't one of them.

Another one that comes up a lot is the idea that "AGI ruin is vanishingly unlikely, but it would be so bad that we should be worried about it anyway". I don't think I've ever seen anyone make that argument with a straight face. Yudkowsky himself thinks that AGI ruin is very likely indeed. (As in, it has a >50% chance of ending life on Earth within a century.)

Of course that group does hold many wacky beliefs. Things like the entire universe splitting into pieces billions of times a second, people's brains being a kind of generalized refrigerator, the wisdom of not taking a free $1000, and the possibility of bringing sufficiently well preserved dead people back to life. Also, thinking that there's a high chance of AGI ruin is, if anything, an even wackier belief than thinking it's very unlikely. So there's still plenty of room to make an argument that Yudkowsky and many of his readers have ended up believing wacky things despite their disdain for religion.

I do find it very odd, though, that those two misconceptions are so popular. It's like Gell-Mann amnesia: If people can be so wrong about this particular internet subculture that I happen to know something about, how can I trust anything said by anyone about a culture they aren't a member of?

[ModernMech]

I didn't say the community at large believes in it. I did say that it's a repackaged idea that originated from their community, and that's 100% true.

Maybe you're interpreting when I said they could "get behind" the idea as to mean I was implying that the entire community endorsed it, but that's not what I had intended to convey. What I meant to say was that some members of that community came up with this idea and then engaged with it in a way they would not engage with God or Jesus and Hell solely because of the framing of the deity as a technology instead of a spirit, and Hell as a simulation instead of another dimension.

[freen]

Classic Chesterton’s Fence.

https://fs.blog/2020/03/chestertons-fence/