[bitwize]

The AI "movement" is hermetic magick. The goal is to bring about God in silico, because if you're not involved in so doing, God may punish you for eternity when he emerges:

https://en.wikipedia.org/wiki/Roko's_basilisk

Next to the might and terror of the machine God, mere humans are, individually, indeed as nothing...

[ben_w]

Most of the people working on AI, and even those on the specific sub-domain of AI where Roko's basilisk was coined which isn't the majority of the field by a long shot, have been rolling their eyes at Roko's basilisk since the moment it was coined.

Even a brief moment of thought should reveal that, even if you think the scenario likely, there are an infinite number of potential equivalent basilisks and you'd need to pick the correct one.

I'm less worried about Roko's basilisk*, and rather more worried about the people who say this:

  I think you have said in fact, and I'm gonna quote, development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity. End quote. You may have had in mind the effect on, on jobs, which is really my biggest nightmare in the long term.

- https://www.techpolicy.press/transcript-senate-judiciary-sub...

Because this is clearly not taking the words themselves at face value; either you should dig in and say "so why should we allow it at all then?" or you should dismiss it as "I think you're making stuff up, why should we believe you about anything?", but not misread such a blunt statement.

(If you follow the link, Altman's response is… not one I find satisfying).

* despite the people who do take it seriously, as such personalities have always been around and seldom cause big issues by themselves; only if AI gets competent enough to help them do this do they become a problem, but by that point hopefully it's also competent enough to help everyone stop them

[salawat]

>only if AI gets competent enough to help them do this do they become a problem, but by that point hopefully it's also competent enough to help everyone stop them

Tell me something; have you ever built something you later regret having built? Like you look back at it, accept you did, but realize that if you'd just been a bit wiser/knowledgeable about the world you wouldn't have done it? In the moment you're doing the thing you'll regret, you don't know in that moment anything better to do until the unpleasant consequences manifest, granting you experience.

If you haven't experienced that yet; fine, but we shouldn't be betting on existential problems with "hopefully" if we can at all avoid it. Especially when that hopefully clause involves something we're making the decision to craft, with means and methods we don't fully understand/aren't predictively ahead of, and knowing that the way these methods work have a tendency to generate/provide the basis to generate a thoroughly sycophantic construct.

[ben_w]

Sure.

To your point, my P(doom) is 0.1, but the reason it's that low is that I expect a lot of people to use sub-threshold AI to do very dangerous things which render us either (1) unwilling or (2) unable to develop post-threshold AI.

The (1) case includes people actually taking this all seriously enough, which as per your final paragraph, I agree with you that people are currently not.

Things like Roko's basilisk are a strict subset of that 0.1; there's a lot of other dooms besides that one.

[cedws]

Sci-fi mumbo jumbo.

[bitwize]

Sci-fi mumbo jumbo that Dario, Sam, and the rest have read and were profoundly influenced by. Yudkowskianism is marked by a strong apocalyptic "evil AI will kill us all" streak, and a big part of the AI industry is a race to build "good" AI before "evil" AI gets a fighting chance. Of course that all goes out the window just like Google's "don't be evil" once advertisers and the Pentagon start wafting the scent of money into the air...