[marvin]

There is nothing religious with thinking about whether failure modes of advanced artificial intelligence can permanently destroy large parts of the reality that humans care about. Just like there was nothing religious in thinking about whether the first atomic bombs could start a chain reaction that would destroy all life on Earth.

Part of such precautionary planning involves asking whether such an accident could happen easily or not. There certainly isn't consensus at the moment, but the philosophy very clearly favors a cautious approach.

Most people are used to thinking about established science that follows expected rules, or incremental advances that have no serious practical consequences. But this isn't that. There is good reason to think that we're approaching a step-change in capabilities to shape the world, and even a strong suspicion of this warrants taking serious defensive measures. Crucially for this particular instance of the discussion, OP is favoring that.

There will necessarily be a broad spectrum of opinions regarding how to handle this, both in the central judgement and how palatably the opinion itself is presented. Using a dismissive moniker like 'religious' for a whole segment of it doesn't give justice to the arguments.

Present a counterargument if you feel strongly about it, and see whether that will stand on its own merit.

[astrange]

The post, which calls Eliezer a "prophet" who says people should drop everything in their lives to work on AI safety, agrees with me.

> Present a counterargument if you feel strongly about it, and see whether that will stand on its own merit.

This is a bad way to talk to rationalists because it's what they think solves everything and is the reason they're convinced an AI is going to enslave them. As long as you're actually right, saying "no that's dumb and not worth worrying about" is superior to logical arguments about things you can't have logical arguments about (because there are unenumerable "unknown unknowns" in the future). This is called "metarationality".

e.g. Someone could decide to kill you because they don't like one of your posts (1). Is there any finite amount of work you could do to stop this? No (2). Should you worry about this? No (3).

You can't logically prove the 2->3 step, nor can you calculate the probability of it being a problem, but it still doesn't seem to be a problem.