[stared]

I have heard it mostly from the Less Wrong crowd.

I mean, all stories about religious dedication to "alignment", with doomsday vision if we do it wrong, and a vision of paradise if we do it correctly.

In particular, the concept of the Roko's Basilisk is some rehash of the Pascal's Wager.

[Jach]

Would you care to share any links? Specifically any talk of an AI-God "judging". If your entire perception is based on what you think you know about the Roko incident, may I suggest that you are ill-informed about it? It wasn't something the "LW crowd" endorsed or believed. Looks like I left some links to help someone else become informed several years ago: https://news.ycombinator.com/item?id=18982933

[stared]

For starter: https://www.lesswrong.com/w/paperclip-maximizer-1

[Jach]

I can see how talk of a paperclip maximizer, or any sort of AGI that can actually deliver on e.g. Drexler's nanotech or otherwise act in very powerful ways like, to quote, "colonizing the galaxies", pattern-matches to something roughly omnipotent by most of our standards. I can see how it roughly pattern-matches to something like "immortal", but this seems like the least of feats, much of our complex machinery with maintenance is effectively immortal already. Overall I don't see how this connects to what you originally wrote: "There is a common narrative bias to look at AGI as the Abrahamic God, if not explicitly, then just by saying that it is omniscient, omnipotent, immortal - and will judge us for our deeds." It is not omniscient, it only knows enough to develop the tech to enable paperclip maximization. It is not judging, either: "The AI does not hate you, nor does it love you, and you are made of atoms it can use for something else."

I should also note that the paperclip maximizer is not something the LW crowd believes should exist. Its primary function as an idea is to illustrate the orthogonality thesis: that goals and intelligence aren't dependent on each other. Its secondary function is to illustrate instrumental convergence.

[wamatt]

> In particular, the concept of the Roko's Basilisk is some rehash of the Pascal's Wager.

Oh boy couldn't agree more. The whole Basilisk drama really caused me to rethink the idea that a group of Rationalists were in practice, were in fact particularly rational. CF The Atheists vs someone that happens to be atheist. To their credit, they also acknowledged this somewhat and hence the CFAR crowd.

The word "rehash" also resonates. Like Google has a tendency to use PHD's to reinvent everything over and make up new words and terms for the same concepts.

Having spent a little too much time in the early 2010's at Bay Area LW meetups, the more fantastical LARP of fanfic, and religious mythology often felt a bit at odds. There was the aspect of a charismatic, autodidact leader obsessed with a certain J.K. Rowling IP and the kinky stuff..Don't get me wrong, I still have fond memories of this time overall :)

From my perspective, a core issue seemed to be no-one seemed to particularly motivated in defining what it meant to be rational, aside from some loose segmentation around instrumental vs epistemic rationality. (ie practice vs theory). And because of this it almost had a faith vibe to the scene. Like "trust me bro" this is "super high brow nerd stuff" and on your third helping of "The Sequences" all these formulas and shiny new words will all make sense what and it will be clear why we’re doing these meetups. (It totally wasn't anything to do with mental masturbation and high-iq crowd bonding and feeling good ;)

When I was into Christian apologetics (C.S. Lewis etc) as in my first year of CS & philosophy of science in college, there was a similar thing.. after a year of seeking out the scientific, and logical explanations for all the religious dogma I was indoctrinated into growing up. In the end, it pretty much reduced to "just have faith". This was after exhausting the "well you're not an expert on Christian theology, so you cant have a solid argument around the nature and existence of God because you need to study more" counter. This despite reading and studying the Bible at length.

For example, right now. Is it rational to be typing this up on HN, when I have other more important goals to do? OTOH reminiscing on the past and connecting with a single serving online friend OP, gives a bit of a dopamine hit. And maybe sharing resonates with others and increases happiness in the world? (or not if anyone still LW reads this and maybe feels a different type of way)

So that’s community right and good (in the sense it's aligning with goals)? But then its driven by emotions, so that kinda is not supposed to be rational. Is it rational to observe one's own mental states and take action? Turtles all the way down!