[johann28]

This is a very important point. His cult can wreck your thinking. I think it should only be read by people with enough contextual information to call his bullshit. He definitely is a smart guy in a way, and you can't easily attack his writings in a way that would be easy to understand for a newcomer.

So I won't attempt that here. Instead, I'll tell my (random internet guy's) feelings and vague ideas: You know when you meet a stranger and you feel something is off, but you don't really know what, but something's funny? Martial art and personal defense coaches usually recommend that you listen to this feeling and act accordingly.

Singulitarianism is basically an apocalyptic doom-religion. It follows the same scheme. For example, I just read a reddit thread (https://www.reddit.com/r/DebateReligion/comments/3vsoe7/chri...) the other day and the post reminded me of Singulitarianism a lot. There is an apocalypse, some external information that you should rely on instead of you intuitive faculties because they are biased and unreliable, but if you follow the path of the leader you'll be saved.

Also, its followers react to criticism similarly to religious people.

And another problem is that nobody serious has ever rolled up their sleeves, went through and criticized the whole mess of their Bible (called Sequences), a massive collection of writings, interlinked thousand-fold in an intricate complex network. Simply because the people who could do this don't care much about some random guy posting stuff to his blog.

Fortunately since the Basilisk story came out and we could see Yudkowsky's reaction, it was an eye-opener for many people. Still his movement is quite well-spread and gets linked to often by CS people, mathematics students etc.

And I won't start to refute individual claims. I admit this. The thing is not about the individual claims but the way they are linked and the kind of narrative it builds. The untold assumptions lurking in the background etc. The way he tries to take credit for age-old ideas by renaming them and not caring about the origin of them, since the history of philosophy is irrelevant anyway, we should just read his blog posts and related stuff.

My advice for novices is to start with the mainstream. You can of course criticize the mainstream once you have sufficient knowledge. Read textbooks, classical works, go to universities, take interesting courses etc. Don't teach yourself this stuff from a random guy's blog who has no qualifications on anything he speaks about (which isn't a disproof but a red flag).

I've been vague and I stand by it. Yes I might have been irrational in my comment or led by feelings and emotions. I don't care. I just had to put this out here. I don't aim my paragraphs as ultimate refutation but as a warning. I felt I had to write this for the benefit of people who may be new to this man's world.

[yummyfajitas]

Climate science is basically an apocalyptic doom-religion. It follows the same scheme. For example, I just read a reddit thread (https://www.reddit.com/r/DebateReligion/comments/3vsoe7/chri...) the other day and the post reminded me of climate science a lot. There is an apocalypse, some external information that you should rely on instead of you intuitive faculties because they are biased and unreliable, but if you follow the path of the leader you'll be saved.

And another problem is that nobody serious has ever rolled up their sleeves, went through and criticized the whole mess of their Bible (called Climate Science Papers), a massive collection of writings, interlinked thousand-fold in an intricate complex network. Simply because the people who could do this don't want to be criticized in the media.

Fortunately since the Climategate story came out and we could see the leadership's reaction, it was an eye-opener for many people. Still this movement is quite well-spread and gets linked to often by journalists, politicians, etc.

None of your critiques are remotely specific to lesswrong. All could be equally well applied to climate science or a variety of other things where I suspect you'd be unwilling to apply it.

[johann28]

If climate science was advocated by a single person without any formal qualification in climatology, physics, hydrology, Earth Science etc. it would be similar.

But Climate science is a lot less about an Apocalypse. Mainstream climate science isn't about a total wiping out of humanity but a few degrees of change in temperature, extreme weather, more deserts, floods etc., which are terrible catastrophes but nowhere near on the scale that Yudkowsky likes to dream up (evil robots destroying everything and torturing and extorting and creating myriads of simulations of simulations of you being tortured in the worst ways possible in various hypothetical, counterfactual scenarios to acausally motivate you to serve these evil overlords etc.)

Also, climate science doesn't ask you to "obviously" dismiss reputable fields and past sources. It's just a very bad analogy. The best defense of Singulitarianism is the "but what if you're wrong" kind of betting. Because their imagined doom scenario is so extremely unimaginably bad, they argue that they can get away with little proof, since multiplying the huge disaster with a little probability still gives a large expected risk. They like to say "shut up and multiply" as a slogan for this, i.e. multiply probability and outcome to get the expectation. He lays the groundwork and path for this such that if you gradually get into his system, he can take you in the woods without you noticing. It's no less than a panic-inducing mind virus.

And yeah, you can say I didn't disprove him with this. I don't want to. My main point is: don't start with his writings and don't recommend them to people who are uneducated in these topics. You'll do them a favor by recommending reputable sources.

[yummyfajitas]

Mainstream climate science isn't about a total wiping out of humanity but a few degrees of change in temperature, extreme weather, more deserts, floods etc.,...

You are sounding like a climate skeptic.

What does Yudkowsky ask you to "obviously dismiss"?

Near as I can tell, your only critique of Yudkowsky is that he asks you to consider various philosophical and scientific edge cases that are outside of the social mainstream, and are therefore somehow wrong in a way you refuse to discuss.

[Aljik]

Your writing implies you think Yudkowsky believes/promotes the Basilisk. Are you not aware he thinks that is nonsense?

[johann30]

I have no idea what he truly believes in and I don't even care too much. What I care about is the effects of his writings. Remember we are debating whether his writings are to be recommended to someone who is new to philosophy about science, algorithms, cognitive biases etc.

The Basilisk is an organic outgrowth from it. And it's just one of many. They follow naturally from the tenets of the religion, they aren't outliers or "bad apples".

[Aljik]

>I have no idea what he truly believes in and I don't even care too much.

It is relevant. Look what you wrote:

>Yudkowsky likes to dream up (evil robots destroying everything and torturing and extorting and creating myriads of simulations of simulations of you being tortured in the worst ways possible in various hypothetical, counterfactual scenarios to acausally motivate you to serve these evil overlords etc.)

Unless I am mistaken, this is a clear reference to Roko's basilisk. EXCEPT Yudkowsky didn't dream it up. Roko did.

Can you elaborate on your issues w/ the Basilisk? As far as I am concerned its just a thought experiment. If Yudkowsky had actually tried to get people to give him money or something via it you would have a good complaint... but he didn't.

[johann28]

> Unless I am mistaken, this is a clear reference to Roko's basilisk. EXCEPT Yudkowsky didn't dream it up. Roko did.

Correct, I was conflating things. But his unfriendly AI scenario is pretty bad too.

My issue with it is that it's a symptom of an over-confident belief system naturally leading people to such conclusions. And now I don't want to get into refuting it, it's been done numerous times to varying degrees. I think the whole underlying system is problematic, there is no superficial mistake in the reasoning, once you accept a few philosophical standpoints and value judgments that the reader is spoon-fed while reading his blog posts.

[yummyfajitas]

To repeat this idea with a different mood affiliation:

I have no idea what muslims truly believe and I don't even care too much. What I care about is the effects of their beliefs...Terrorism is an organic outgrowth from it. And it's just one of many...

I take it that if Donald Trump makes this claim, you'll support it?

[pron]

How are "muslims", over 1.5 billion people with wide ranging beliefs and practices analogous to one individual who leads a small group? If you were to apply this to a particular muslim with a following, who you believe has advanced particular beliefs with a negative effect, then the sentiment expressed by johann30 would make a lot of sense, and even the most heart-bleeding of liberals would agree.

I don't have enough information to agree or disagree with johann30 specific claim, and I can't comment to the negative effects he says he had witnessed on himself and others (all I know is that Yudkowsky is neither a philosopher nor an expert on the subject), but your analogy did not resemble his idea at all.

[dang]

Please stop trolling on HN.

[johann28]

Seems like I can't comment normally anymore due to downvotes or some other mechanism. So I quit the discussion now.

[tangled_zans]

I had the same issue and I've just seen this:

https://www.quora.com/Hacker-News-How-come-I-cant-reply-in-a...

I think that's a pretty good lesson from PG!

[sawwit]

New users quickly reach a comment limit. You can create a new account on a different IP and continue.

[gglitch]

I feel you. I'm reminded of Robert Kegan's stages of development, whereof the rare individual who reaches stage four, the stage where one learns to transcend irrational personal attachments and instead cultivate logical, systematic principles and practices to guide behavior, nevertheless sooner or later starts to find niche problems or, let's say, glitches within his or her system, and hopefully is able to eventually reach a stage five, wherein one is able to move among systems freely, picking them up and setting them down as needed, because the whole damn thing, the human situation, is funny and crooked and lovely and not a neatly determined algorithm. Kegan's is of course an overdetermined system. It has problem spots. It can be picked up and set down. It's funny. Lesswrong on the other hand, I personally haven't yet found the part where someone shrugs and says, Hey, this works pretty well for me, but it won't yield you an e.e. cummings poem to appreciate on a snowy day. It feels like the beginning of an effort to evolve into something like that mysterious race of human computers who own the monopoly on space travel in Dune.

[johann28]

In my time at LW I saw so many people becoming anxious, struggling with how to "escape themselves", how to transcend their biases, and meta-biases, knotting themselves into paradoxes with the elaborate hypothetical constructions, it was just sad after some point. They convinced themselves that rationality means trusting an external system so that if it leads to something counterintuitive, it must surely be seriously accepted. I mean it's kind of how it works, you shouldn't rule out your potential conclusions at the outset, but real life is a lot messier than theories. And I know one can say, this sounds like an excuse to be lazy and unreflective and uninterested in the world's problems, but actually all I'm saying is you can't obsess over this sort of thing and keep your mental hygiene at the same time. This sort of thing can make people burn out or become actually depressed or paranoid.

One must keep the ability to laugh at oneself, to be able to humorously see a kind of futility in what one does but dance the dance anyway. Clenching too hard, making one more nested iteration in the prisoner's dilemma or the chicken game won't help.

Also don't take stuff more seriously than a certain ceiling.

For me, reading about Zen ideas helped me get out of this narrow, "rationalized", neatly ordered, algorithmic, packaged-and-labeled way of thinking. Metaphorically it's kind of the difference between a probabilistic machine learning system vs. a symbolic knowledge system where everything is defined precisely and unambiguously and every rule is laid out etc.

I'm still materialist and atheist, but I think being too deeply involved in any ideology is harmful (be it Marxism or Fascism or Scientology or LW). Yeah, you can reflect upon whether it's possible to live truly independent of ideologies without making or finding a new one for yourself. But there are certain indicators, like when you feel you're getting distanced away from the people around you physically, when you start feeling superior for belonging to the in-group etc., it usually means you're just not noticing some aspects and are obsessing over something. Sure one can say that if innovators thought like this and always stayed in line of the mainstream, we never would have gotten Ford or Jobs etc. And there is truth to this, but it kind of sounds like "X dropped out of college and went on to be successful therefore dropping out of college is a good idea". No, generally the good idea is to be humble and positive about the people around you, be open and reflective but not obsessive and be able to relax.

[tangled_zans]

It's sound advice, I'd second it.

It's not so much that we "hate" Elizer, like Aljik suggests, it's just that on the whole - compared to the many mindblowing authors mentioned in this thread - Elizer just isn't particularly outstanding. Someone has to say it, lest anyone reading this for the first time gets the wrong impression.

Anyone's welcome to still read him, for all we care, but you may as well be getting your philosophy education from Cosmopolitan.

[johann28]

> Anyone's welcome to still read him, for all we care,

Of course, but I'd qualify this a bit. If you are in a life situation where you feel alone or in need of a community, you feel you're not included enough, or feel "smarter" and more reflective than your environment, then spending time at LW can amplify your smugness and stroke your ego to the point of not noticing what you are turning into. I saw many people there with milder psychological problems. Really, a psychologist would probably have an interesting time analyzing the "life advice/coaching/coping" parts of that forum.

However, if you're living a balanced, good life, reading this stuff won't matter much. It's similar in many ideologies. If you're in a receptive situation, then Scientology, or Reddit's Redpill, or basically any elitist, enlightening type of subculture along the lines of "you now understand what it all really is about" can be harmful.

[tangled_zans]

Yeah, agreed! It's no surprise that LW has been known to attract a few PUA followers as well.

[Aljik]

> Someone has to say it, lest anyone reading this for the first time gets the wrong impression.

Sorry what? You have to say that isn't outstanding, unless some one gets the impression he is?

Maybe this is crazy idea, but if Elizier wasn't outstanding... wouldn't that be something people would figure out for themselves? On the otherhand if people are reading his works, and walking away with the impression that he is outstanding, couldn't it simply be that it is? And even so, who are you to say that is wrong?

On the other hand, if there really are authors who write about similar things,but better than Yudkowsky I would be interested in reading (I haven't found any).

So, since you claim to know of such things, could you pick something Yudkowsky has written about and name me another who does it better?

[tangled_zans]

Who am I? WHO AM I?

I am some random anon posting their views on the internet, as is every other person here. It's called having a conversation.

"So, since you claim to know of such things, could you pick something Yudkowsky has written about and name me another who does it better?"

Yes, I could. But since you've so far been a total arrogant assface, and since your tone implies that - rather than a genuine interest - you simply want to bait me into a further argument so you can show off how much you're in love with Elizer... I'm going to pass. Do your own research if you're so "interested".

[Aljik]

>Yes, I could. But since you've so far been a total arrogant assface, and since your tone implies that - rather than a genuine interest - you simply want to bait me into a further argument so you can show off how much you're in love with Elizer... I'm going to pass.

For a conversation you being pretty insulting. But, no, contrary to your assumption its actual curiosity.

If you are worried about this argument continuing if you refer me to such works, I promise not discuss them if you post them.

[Aljik]

Could you elaborate? How can it wreck one's thinking? I mean, maybe I'm wrong but your writing as if it is dangerous to read for the untrained to read.

I agree with singularity being like a religion. I think the singularity is nonsense. On the otherhand I don't think the rest of your accusations are accurate.

First, I'm not sure I would name Yudkowsky the leader of the singularity culture. Of lesswrong certainly, but lesswrong is dead. Yudkowsky is about as much a leader of the singularity stuff, as P.G. is of "hacker" culture. A big name with writings on philosophy and the culture. Honeslty I would name Kurzweil as a far bigger person w.r.t singularity stuff. He is far more known.

Furthermore no where does Yudkowsky say anything like "if you follow me you'll be saved". In fact he probably believes the singularity will save everyone (if friendly a.i).

This is where I expect you to bring up Roko's Basilisk. For those unaware, Roko's Basilisk is the idea that a A.I may retroactively punish people who didn't assist in it's development. Therefore if you don't want to suffer you should donate money and time to creating A.I.

First of all, this was idea was created by a guy called, Roko. Not Yudkowsky. Yudkowsky doesn't believe it. Yudkowsky's response among other things was to say that it was not ethical of Roko to write about the basilisk, if Roko really believed it to be true.

Kind of like how sometimes atheists will say that Christians shouldn't proselytize to people who have never heard of Christianity, because if they don't know about it, God can't very well send them to hell for their disbelief.

Anyways I am interested to hear why you felt so strongly you had to warn us off him. The hate Yudkowsky gets is something I don't understand but want too.

[tangled_zans]

I have literally just brought up Roko's basilisk in a different branch of this thread! Cray cray.

[johann30]

I was gullible enough once to eat up his philosophy. Because he's not trivially wrong. He tries to take credit for developments that aren't his, present good ideas mixed in with nonsense and you just swallow it all naively, unless you've previously thought through the stuff. His casual dismissing of the mainstream can become your style as well, and you can feel you become part of something special, something that transcends most people's levels. It's really not unlike Scientology.

Sorry if I come across as warning too much. It probably depends on your personality type. If you are a very reflective and self-critical person, you can tie yourself up with his philosophy I can tell you that much. Now, granted, the consequences of something don't tell much about the truth of that thing (just like the "God exists because if God exists then I feel safe and happy and gives meaning to my life" is a bad argument.), still - keeping in mind that we are discussing whether a novice should be advised to read Yudkowsky - I think there are better choices.

I absolutely think that if you're above average smart but lack the factual, lexical knowledge, then reading his stuff can be detrimental to your intellectual development. You're better off without it. Yeah, you can make the case for going through such experiences, just like you can make the case that going through drug addiction and recovery can make you a better/stronger person in some sense, but as a first approximation it's better to stick with mainstream literature. That's the main point I'm trying to argue for.

And I have to add that it's weird and frustrating to argue about this because obviously those defending his ideology are smart enough to come up with good counterarguments. It's nothing like debating with, say, Young Earth Creationists. These people (perhaps including you) are often intelligent, tech/math/CS-literate people. The point of disagreement is of a finer nature that is hard to even discuss because that whole aspect is usually dismissed as "useless philosophy".

It's somewhat akin to https://xkcd.com/793/ where someone who masters one technical field feels an immense power or superiority and becomes a bit smug.

Feynman had the cred to be able to say "If you don't like it [the way nature works], go somewhere else; to another universe maybe", but talking in this kind of confident way requires a very solid background. I'm not talking about prizes, simply actual results. And I know it's difficult due to the nature of the topic (studying a hypothetical unknown-probability event that can wipe out everything), but this sort of cop-out is just not convincing.

[jimrandomh]

> His casual dismissing of the mainstream can become your style as well, and you can feel you become part of something special, something that transcends most people's levels. It's really not unlike Scientology.

Um. You might want to read up on what Scientology is; they're not even remotely in the same league. For example, Yudkowsky has never been accused of war crimes. Also, his brainwashing technique seems to be a bunch of essays, rather than, say, a biofeedback device coupled with personalized verbal abuse and a systematic dismantling of peoples' personal relationships.

[johann28]

The becoming part of a special in-crowd who will save the world is the similarity. Not the torture and crimes of course. I've seen and read quite a lot about Scientology, but you are right that they are nowhere near the same league. Scientology is a massive and extremely rich bunch, who do some very bad things on purpose.

[tangled_zans]

>It's somewhat akin to https://xkcd.com/793/ where someone who masters one technical field feels an immense power or superiority and becomes a bit smug.

That's what really put me off of the LW community as a whole - that condensing attitude of superiority over anyone not in the in-group. There was a telling comment in one of the articles that went along the lines of: "Oh yeah, LW is not for everyone. It takes a particular kind of person to discuss the intricacies of the Newcomb's problem day in and day out". That made me cringe so much when I read it. Like, jeez, just get over yourself, no one cares how "smart" you are.