[gwern]

That's not interesting at all, because it was written partially to target Yudkowsky, with the goal that you won't "Pascal's mug"* yourself eg https://twitter.com/sashachapin/status/1657063187655843841

Keep in mind that a lot of tech-adjacent writing these days is just AI debates (and this is why he is not engaging with the actual cult literature or providing examples); this is not the place to debate the object-level arguments, but I will say I disapprove of Chapin writing like this, and not owning up to the real purpose of this essay and the Bay Area dynamics he's criticizing. It is deceptive in precisely the way you inadvertently illustrate.

* His use is wrong, incidentally, both in the original abstruse decision-theory sense of the phrase as coined by Yudkowsky, ironically enough, and in the vulgarized sense of 'you should ignore small probabilities of very bad things' (because we are now far beyond some 'small' probability of AI, and AI risk is now considered so probable people like Geoff Hinton are quitting their jobs so they can speak out about it https://www.lesswrong.com/posts/bLvc7XkSSnoqSukgy/a-brief-co... )

[sashachapin]

Gwern, this seems like mind-reading. The real purpose of this essay was to muse about a set of character traits I find interesting, not participate in an AI debate. Most of the people I was thinking of are not involved in AI. (I'm much more a part of the meditation world than the AI world, and there's a LOT more cultiness there; and a more thorough survey of my twitter would have revealed that I think about meditation more than AI.)

Yudkowsky was definitely a case I thought of, but not the specific target of this article. AND, Yud is actually an example of one of these personalities that I think is probably net positive! Even if I have a lot of objections to the way he's presented himself and his specific arguments, I think he's doing a good job with moving the Overton window of taking AI seriously.

I grant that I probably do not fully understand Pascal's Mugging. But this tweet was a subtweet of someone who said that she was working on AI because an aligned AI would definitely end factory farming; whether or not this is true, this seems like the kind of thinking that will drive people crazy (all concerns must be suborned to the One Great Cause.)

[alexb_]

I'm saying more that it's "interesting" in an argumentative way, not that it's actually of interest about how this is about EY. It should be interesting to followers of EY because all of these traits are things that cult leaders do, and are things that EY does, because EY is (intentionally or not) running a doomsday cult. It's impressive how much reach he's been able to get, to the point where his ideas of control and morality have seeped into the most important technology of our time, but it's still a cult founded on shaky conjecture nonetheless. I think it's pretty clear if you are interested in AI who this article is about, although I do agree it should have been much more explicitely followed up with "Here's how EY is these things".

It's made me very sad to see extremely smart people, who I once looked up to and really found very interesting (you're included in here!) fall victim to the irrational fear of AI spurred on by Eliezer. You can create as much literature as you want, and you can create as many hypothetical scenarios as possible, but it's not going to change the fact that dedicating your life into controlling AI so that you can "logically" justify your own life is nonsense.

It's not that I don't get it. I just disagree. I don't think AI will create a science fiction horror show to end all of humanity. I don't think AI will cause humans to stop being humans. I don't think AI will create the end of the world. Yes, I am not 100% sure. There is a 0.01% chance of this all happening. But that doesn't mean that we need to pay attention to it just because multiplying 0.0001 by 1,000,000 yields a big number. I think basing your morality and existence on trolley problems and mental experiments is abhorrent. And no, I can't give you an exact spelled out reason as to why. Do I have to justify basic human morality?

You're an extremely smart person. I love your blog posts, and I love the depth and time you put into them. It's incredible how disappointing it is for me to see you become another "rationalist" obsessed with a fantasy of AI doom.

Edit for your edit: Someone who is wealthy and has nothing to lose quitting his job to join the cult of AI doomerism is, in no way, proof that AI xrisk is meaningful. You're using someone else coming to a conclusion as a reason you are correct. I trust you are smart enough to know why this is awful logic.

[gwern]

> It should be interesting to followers of EY because all of these traits are things that cult leaders do

It's not really because it's glib pattern-matching. It's roughly "Is the NIH a cult?" https://www.michaeleisen.org/blog/?p=1217 level. Go compare Eliezer to say, NXIVM, to see what a real cult looks like.

> It's made me very sad to see extremely smart people, who I once looked up to and really found very interesting (you're included in here!) fall victim to the irrational fear of AI spurred on by Eliezer. You can create as much literature as you want, and you can create as many hypothetical scenarios as possible, but it's not going to change the fact that dedicating your life into controlling AI so that you can "logically" justify your own life is nonsense...You're an extremely smart person. I love your blog posts, and I love the depth and time you put into them. It's incredible how disappointing it is for me to see you become another "rationalist" obsessed with a fantasy of AI doom.

I feel I should point out that you have not seen me 'become another rationalist' because this is what I have been since around 2004, long before I started any writing or even was 'gwern'. 'AI doom' has always driven my writing (even if I sometimes wander quite far afield before eg. concluding that nootropics are a dead end for intelligence augmentation that might help with AI risk - on the bright side, I think DL scaling may have helped answer why nootropics have been such a bust in general). I just thought we'd have way more time.

> Someone who is wealthy and has nothing to lose quitting his job to join the cult of AI doomerism is, in no way, proof that AI xrisk is meaningful. I trust you are smart enough to know why this is awful logic.

'some wealthy dude' is not the relevant description here for Geoff Hinton. I trust you are smart enough to see the problem with your even glibber comment there.

[rolisz]

Yes, a 100 times this! Thank you for putting this so nicely into words! Too bad there's no HN gold or whatever for me to award you!