[samuel]

I'm currently reading Yudkowsky's "Rationality: from AI to zombies". Not my first try, since the book is just a collection of blog posts and I found it a bit hard to swallow due its repetitiveness, so I gave up after the first 50 "chapters" the first time I tried. Now I'm enjoying it way more, probably because I'm more interested in the topic now.

For those who haven't delved(ha!) into his work or have been pushed back by the cultish looks, I have to say that he's genuinelly onto something. There are a lot of practical ideas that are pretty useful for everyday thinking ("Belief in Belief", "Emergence", "Generalizing from fiction", etc...).

For example, I recall being in lot of arguments that are purely "semantical" in nature. You seem to disagree about something but it's just that both sides aren't really referring to the same phenomenon. The source of the disagreement is just using the same word for different, but related, "objects". This is something that seems obvious, but the kind of thing you only realize in retrospect, and I think I'm much better equipped now to be aware of it in real time.

I recommend giving it a try.

[Bjartr]

Yeah, the whole community side to rationality is, at best, questionable.

But the tools of thought that the literature describes are invaluable with one very important caveat.

The moment you think something like "I am more correct than this other person because I am a rationalist" is the moment you fail as a rationalist.

It is an incredibly easy mistake to make. To make effective use of the tools, you need to become more humble than before you were using them or you just turn into an asshole who can't be reasoned with.

If you're saying "well actually, I'm right" more often than "oh wow, maybe I'm wrong", you've failed as a rationalist.

[zahlman]

> The moment you think something like "I am more correct than this other person because I am a rationalist" is the moment you fail as a rationalist.

Well said. Rationalism is about doing rationalism, not about being a rationalist.

Paul Graham was on the right track about that, though seemingly for different reasons (referring to "Keep Your Identity Small").

> If you're saying "well actually, I'm right" more often than "oh wow, maybe I'm wrong", you've failed as a rationalist.

On the other hand, success is supposed to look exactly like actually being right more often.

[Bjartr]

> success is supposed to look exactly like actually being right more often.

I agree with this, and I don't think it's at odds with what I said. The point is to never stop sincerely believing you could be wrong. That you are right more often is exactly why it's such an easy trap to fall into. The tools of rationality only help as long as you are actively applying them, which requires a certain amount of humility, even in the face of success.

[wizzwizz4]

Chapter 67. https://www.readthesequences.com/Knowing-About-Biases-Can-Hu... (And since it's in the book, and people know about it, obviously they're not doing it themselves.)

[FeepingCreature]

Also the Valley of Bad Rationality tag. https://www.lesswrong.com/w/valley-of-bad-rationality

[TeMPOraL]

Also that the Art needs to be about something else than itself, and a dozen different things. This failure mode is well known in the community; Eliezer wrote about it to death, and so did others.

[throwanem]

To no avail, alas. But this is why we now see a thought leader publish a piece to say this is a thing it's now permissible not to be, indeed never to have been at all.

[wannabebarista]

This reminds me of undergrad philosophy courses. After the intro logic/critical thinking course, some students can't resist seeing affirming the antecedent and post hoc fallacies everywhere (even if more are imagined than not).

[the_af]

> The moment you think something like "I am more correct than this other person because I am a rationalist" is the moment you fail as a rationalist.

It's very telling that some of them went full "false modesty" by naming sites like "LessWrong", when you just know they actually mean "MoreRight".

And in reality, it's just a bunch of "grown teenagers" posting their pet theories online and thinking themselves "big thinkers".

[mariusor]

> you just know they actually mean "MoreRight".

I'm not affiliated with the rationalist community, but I always interpreted "Less Wrong" as word-play on how "being right" is an absolute binary: you can either be right, or not be right, while "being wrong" can cover a very large gradient.

I expect the community wanted to emphasize how people employing the specific kind of Bayesian iterative reasoning they were proselytizing would arrive at slightly lesser degrees of wrong than the other kinds that "normal" people would use.

If I'm right, your assertion wouldn't be totally inaccurate, but I think it might be missing the actual point.

[mananaysiempre]

> I always interpreted "Less Wrong" as word-play on how "being right" is an absolute binary

Specifically (AFAIK) a reference to Asimov’s description[1] of the idea:

> [W]hen people thought the earth was flat, they were wrong. When people thought the earth was spherical, they were wrong. But if you think that thinking the earth is spherical is just as wrong as thinking the earth is flat, then your view is wronger than both of them put together.

[1] https://skepticalinquirer.org/1989/10/the-relativity-of-wron...

[mariusor]

Cool, I didn't know the quote, nor that it was inspiration for the name. Thank you.

[bmacho]

It's not even about the quote, or Asimov.

"Less wrong" is a concept that has a lot of connotations that just automatically appear in your mind and help you. What you wrote "It's very telling that some of them went full "false modesty" by naming sites like "LessWrong", when you just know they actually mean "MoreRight"." isn't bad because of Asimov said so, or because you were unaware of a reference, but because it's just bad.

[the_af]

> I'm not affiliated with the rationalist community, but I always interpreted "Less Wrong" as word-play on how "being right" is an absolute binary: you can either be right, nor not be right, while "being wrong" can cover a very large gradient.

I know that's what they mean at the surface level, but you just know it comes with a high degree of smugness and false modesty. "I only know that I know nothing" -- maybe, but they ain't no modern day Socrates, they are just a bunch of nerds going online with their thoughts.

[mariusor]

Sometimes people enjoy being clever not because they want to rub it in your face that you're not, but because it's fun. I usually try not to take it personally when I don't get the joke and strive to do better next time.

[the_af]

That's mildly insulting of you.

I do get the joke; I think it's an instance of their feelings of "rational" superiority.

Assuming the other person didn't get the joke is very... irrational of you.

[zahlman]

>but you just know it comes with a high degree of smugness and false modesty

No; I know no such thing, as I have no good reason to believe it, and plenty of countering evidence.

[astrange]

Very rational of you, but that's the problem with the whole system.

If you want to avoid thinking you're right all the time, it doesn't help to be clever and say the logical opposite. "Rationally" it should work, but it's bad because you're still thinking about it! It's like the thinking of a pink elephant thing.

Other approaches I recommend:

* try and fail to invest in stocks

* read Meaningness's https://metarationality.com

* print out this meme and put it on your wall https://imgflip.com/i/82h43h

[Matticus_Rex]

So much projection.

[the_af]

I don't think I'm more clever than the average person, nor have I made this my identity or created a whole tribe around it, nor do I attend nor host conferences around my cleverness, rationality, or weird sexual fetishes.

In other words: no.

[greener_grass]

I think there is an arbitrage going on where STEM types who lack background in philosophy, literature, history are super impressed by basic ideas from those subjects being presented to them by stealth.

Not saying this is you, but these topics have been discussed for thousands of years, so it should at least be surprising that Yudkowsky is breaking new ground.

[elt895]

Are there other philosophy- or history-grounded sources that are comparable? If so, I’d love some recommendations. Yudkowsky and others have their problems, but their texts have an interesting points, are relatively easy to read and understand, and you can clearly see which real issues they’re addressing. From my experience, alternatives tend to fall into two categories: 1. Genuine classical philosophy, which is usually incredibly hard to read and after 50 pages I have no idea what the author is even talking about anymore. 2. Basically self help books that take one or very few idea and repeat them ad nouseam for 200 pages.

[wannabebarista]

Likely the best resource to learn about philosophy is the Stanford Encyclopedia of Philosophy [0]. It's meant to provide a rigorous starting point for learning about a topic, where 1. you won't get bogged down in a giant tome on your first approach and 2. you have references for further reader.

Obviously, the SEP isn't perfect, but it's a great place to start. There's also the Internet Encyclopedia of Philosophy [1]; however, I find its articles to be more hit or miss.

[0] https://plato.stanford.edu

[1] https://iep.utm.edu

[aubanel]

I've read Bertrand Russell's "A History of Western Philosophy" and it's the first ever philosophy book that I didn't drop after 10 pages, because of 2 things: 1- He's logic (or at least has the same STEM kind of logic that we use), so he builds his reasoning logically and not via bullshit associations like plays on words or contrived jumps. 2- He's not afraid to tell "this philosopher said that, it was an error", which is extremely new compared to other scholars who don't feel authorized to criticise even obvious errors. Really recommend!

[NoGravitas]

I don't know if there's anything like a comprehensive high-level guide to philosophy that's any good, though of course there are college textbooks. If you want real/academic philosophy that's just more readable, I might suggest Eugene Thacker's "The Horror of Philosophy" series (starting with "In The Dust Of This Planet"), especially if you are a horror fan already.

[voidhorse]

It's not a nice response but I would say: don't be so lazy. Struggle through the hard stuff.

I say this as someone who had the opposite experience: I had a decent humanities education, but an abysmal mathematics education, and now I am tackling abstract mathematics myself. It's hard. I need to read sections of works multiple times. I need to sit down and try to work out the material for myself on paper.

Any impression that one discipline is easier than another probably just stems from the fact that you had good guides for the one and had the luck to learn it when your brain was really plastic. You can learn the other stuff too, just go in with the understanding that there's no royal road to philosophy just as there's no royal road to mathematics.

[sn9]

People are likely willing to struggle through hard stuff if the applications are obvious.

But if you can't even narrow the breadth of possible choices down to a few paths that can be traveled, you can't be surprised when people take the one that they know that's also easier with more immediate payoffs.

[staunton]

When you've read that passage in the math book twenty times, you eventually come to the conclusion that you understood it (even if in some rare cases you still didn't).

When "struggling through" a philosophy book, that doesn't happen in my experience. In fact, if you look up what others thought that passage means, you'll find no agreement among a bunch of people who "specialize" in authors who themselves "specialized" in the author you're reading. So reading that stuff I feel I have to accept that I will never understand what's written there and the whole exercise is just about "thinking about it for the sake of thinking". This might be "good for me" but it's really hard to keep up the motivation. Much harder than a math book.

[voidhorse]

I agree with the phenomenon you are talking about, but for mathematics, beyond calculation, the situation isn't really different (and no wonder since you'll quickly end up in the philosophy of mathematics).

You can take an entire mathematical theory on faith and learn to perform rote calculations in accordance with the structure of that theory. This might be of some comfort, since, accepting this, you can perform a procedure and see whether or not you got the correct result (but even this is a generous assumption in some sense). When you actually try to understand a theory and go beyond that to foundations, things become less certain. At some point you will accept things, but, unless you have enough time to work out every proof and then prove to yourself that the very idea of a proof calculus is sound, you will be taking something on faith.

I think if people struggle with doing the same thing with literature/philosophy, it's probably just because of a discomfort with ambiguity. In those realms, there is no operational calculus you can turn to to verify that, at least if you accept certain things on faith, other things must work out...expect there is! Logic lords over both domains. I think we just do a horrible job at teaching people how to approach literature logically. Yes, the subtle art of interpretation is always at play, but that's true of mathematics too and it is true of every representational/semiotic effort undertaken by human beings.

As for use, social wit and the ability to see things in new lights (devise new explanatory hypotheses) are both immediate applications of philosophy and literature, just like mathematics has its immediate operational applications in physics et al.

[molochai]

At the risk of being roasted for recommending pop-culture things, the podcast Philosophize This is pretty good for a high-level overview. I'm sure there are issues and simplifications, and it's certainly not actual source material. The nice part is it's sort of a start-to-finish, he goes from the start of philosophy to modern day stuff, which helps a ton in building foundational understanding without reading everything ever written.

[ashwinsundar]

I don't have an answer here either, but after suffering through the first few chapters of HPMOR, I've found that Yudk and others tech-bros posing as philosophers are basically like leaky, dumbed-down abstractions for core philosophical ideas. Just go to the source and read about utilitarianism and deontology directly. Yudk is like the Wix of web development - sure you can build websites but you're not gonna be a proper web developer unless you learn HTML, CSS and Javascript. Worst of all, crappy abstractions train you in some actively bad patterns that are hard to unlearn

It's almost offensive - are technologists so incapable of understanding philosophy that Yudk has to reduce it down to the least common denominator they are all familiar with - some fantasy world we read about as children?

[AnimalMuppet]

I'd like what the original sources would have written if someone had fed them some speak-clearly pills. Yudkowsky and company may have the dumbing-down problem, but the original sources often have a clarity problem. (That's why people are still arguing about what they meant centuries later. Not just whether they were right - though they argue about that too - but what they meant.)

Even better, I'd like some filtering out of the parts that are clearly wrong.

[chipsrafferty]

Have you considered that the ideas that philosophy discusses are very complex, and words are not a sufficient medium to describe those ideas?

That's why you have people arguing over what someone meant. Dumbing it down or trying to write something unambiguous doesn't actually make it better.

[AnimalMuppet]

I will grant you that the ideas are complex. But if words are not a sufficient medium, then we can't think clearly about the ideas.

Original philosophers have the right to define their own terms. If they can't define them clearly, then they probably aren't thinking clearly enough about their ideas to be able to talk to the rest of us about them. (Unless they consider it more important to sound impressive and hard to understand. But if that's the case, we can say "wow, you sound impressive" and then ignore them.)

[HDThoreaun]

HPMOR is not supposed to be rigorous. It’s supposed to be entertaining in a way that rigorous philosophy is not. You could make the same argument about any of Camus’ novels but again that would miss the point. If you want something more rigorous yudkowsky has it, bit surprising to me to complain he isn’t rigorous without talking about his rigorous work.

[ashwinsundar]

I really have no interest in reading any more of that guy’s “work”

[HDThoreaun]

Totally fair but why comment on something you arent interested in understanding

[FeepingCreature]

In AI finetuning, there's a theory that the model already contains the right ideas and skills, and the finetuning just raises them to prominence. Similarly in philosophic pedagogy, there's huge value in taking ideas that are correct but unintuitive and maybe have 30% buy-in and saying "actually, this is obviously correct, also here's an analysis of why you wouldn't believe it anyway and how you have to think to become able to believe it". That's most of what the Sequences are: they take from every field of philosophy the ideas that are actually correct, and say "okay actually, we don't need to debate this anymore, this just seems to be the truth because so-and-so." (Though the comments section vociferously disagrees.)

And it turns out if you do this, you can discard 90% of philosophy as historical detritus. You're still taking ideas from philosophy, but which ideas matters, and how you present them matters. The massive advantage of the Sequences is they have justified and well-defended confidence where appropriate. And if you manage to pick the right answers again and again, you get a system that actually hangs together, and IMO it's to philosophy's detriment that it doesn't do this itself much more aggressively.

For instance, 60% of philosophers are compatibilists. Compatibilism is really obviously correct. "What are you complaining about, that's a majority, isn't that good?" What is wrong with those 40% though? If you're in those 40%, what arguments may convince you? Repeat to taste.

[margalabargala]

Additional note: compatibilism is only obviously correct if you accept that "free will" actually just means "the experienced perception/illusion of free will" as described by Schopenhauer.

Using a slightly different definition of free will, suddenly Compatibilism becomes obviously incorrect.

And now it's been reduced to quibbling over definitions, thereby reinventing much of the history of philosophy.

[FeepingCreature]

I think free will as philosophically used is inherently self-defeating and one of the largest black marks on the entire field, to be fair.

[margalabargala]

Why is that?

Here's what we know:

- we appear to experience what we call free will from our own perspective. This isn't strong evidence obviously. - we are aware that we live in a world full of predictable mechanisms of varying levels of complexity, as well as fundamentally unpredictable mechanisms like quantum mechanics. - we know we are currently unable to fully model our experience and predict next steps. - we know that we don't know whether consciousness as an emergent property of our brains is fully rooted in predictable mechanisms or has some decree of unknowability to it.

So really "do we have free will" is a question that relies on the nature of consciousness.

[FeepingCreature]

No, I disagree with this conclusion. The problem is very much solvable if one simply keeps the map/territory split in mind, and for every thing asks himself, "am I perceiving reality or am I perceiving a property of my brain?" That is, we "experience free will" - this is to say, our brain reports to us that it evaluated multiple possible behaviors and chose one. However, this does not indicate that multiple behaviors were physically possible, it only indicates that multiple behaviors were cognitively evaluated. In fact, because any deciding algorithm has to evaluate a behavior list or even a behavior tree, there is no reason at all to expect this to have any connection to physical properties of the world, such as quantum mechanics.

(The relevant LessWrong sequence is "How An Algorithm Feels From Inside" https://www.lesswrong.com/posts/yA4gF5KrboK2m2Xu7/how-an-alg... which has nothing to do with free will, but does make very salient the idea that perceptions may be facts about your cognition as easily, if not more easily, as facts about reality.)

And when you have that view in mind, you can ask: "wait, why would the brain be sensitive to quantum physics? It seems to be a system extremely poorly suited to doing macroscopic quantum calculations." Once the alternative theory of "free will is a perception of your cognitive algorithm" is salient, you will notice that the entire free-will debate will begin to feel more and more pointless, until eventually you no longer understand why people think this is a big deal at all, and then it all feels rather silly.

[TheAncientGeek]

There isn't a single philosophical meaning. You are probably thinking of libertarian free will..furthering Obviously false because determinism isn't obviously true.

[margalabargala]

> And it turns out if you do this, you can discard 90% of philosophy as historical detritus

This is just the story of the history of philosophy. Going back hundreds of years. See Kant and Hegel for notable examples.

[FeepingCreature]

Sure, and I agree that LW is doing philosophy in that sense.

[margalabargala]

Sure, I just object to the characterisation of "actually correct", as though each of those ideas has not gone back and forth on philosophers thinking that particular idea is "actually correct" for centuries. LW does not appear to have much if any novel insight; just much better marketing.

[FeepingCreature]

I think philosophy has gone back and forth for so long that they're now, as a field, pathologically afraid of actually committing.

The AI connection with LessWrong means that the whole thing is framed with a backdrop of "how would you actually construct a mind?" That means you can't just chew on the questions, you have to actually commit to an answer and run the risk of being wrong.

This teaches you two things: 1. How to figure what you actually believe the answer is, and why, and make sure that this is the best answer that you can give; 2. how to keep moving when you notice that you made a misstep in part 1.

[TheAncientGeek]

>actually, this is obviously correct

Nobody know what's actually correct, because you have to solve epistemology first, and you have to solve epistemology to solve epistemology...etc.etc.

>And it turns out if you do this, you can discard 90% of philosophy as historical detritus

Nope. For instance , many of the issues Kant raised are still live.

>The massive advantage of the Sequences is they have justified and well-defended confidence

Nope. That would entail answering objections , which EY doesn't stoop to.

>Compatibilism is really obviously correct

Nope. It depends on a semantic issue , what free will means.

[sixo]

To the Stem-enlightened mind, the classical understanding and pedagogy of such ideas is underwhelming, vague, and riddled with language-game problems, compared to the precision a mathematically-rooted idea has.

They're rederiving all this stuff not out of obstinacy, but because they prefer it. I don't really identify with rationalism per se, but I'm with them on this--the humanities are over-cooked and a humanity education tends to be a tedious slog through outmoded ideas divorced from reality

[biofox]

If you contextualise the outmoded ideas as part of the Great Conversation [1], and the story of how we reached our current understanding, rather than objective statements of fact, then they becomes a lot more valuable and worthy of study.

[1] https://en.wikipedia.org/wiki/Great_Conversation

[jstanley]

But isn't the content of LessWrong part of the Great Conversation too?

[jay_kyburz]

I have kids in high school. We sometimes talk about the difference between the black and white of math or science, and the wishy washy grey of the humanities.

You can be right or wrong in math. You have can an opinion in English.

[bmacho]

You can be right or wrong in math and philosophy. You have can an opinion in any other sciences, physics, chemistry, biology, medical sciences, history, you name it.

[greener_grass]

I suggest teaching them about the Problem of Induction [1].

Scientific thinking is not the same as mathematical thinking and it becomes quite wishy washy grey if you zoom in too far!

[1] https://plato.stanford.edu/entries/induction-problem/

[HDThoreaun]

You can definitely have wrong opinions in the humanities.

[HDThoreaun]

Rationalism largely rejects continental philosophy in favor of a more analytic approach. Yes these ideas are not new, but they’re not really the mainstream stuff you’d see in philosophy, literature, or history studies. You’d have to seek out these classes specifically to find them.

[TheAncientGeek]

Analytical philosophy is rationalism done right.

[TimorousBestie]

They largely reject analytic philosophy as well. Austin and Whitehead are roughly as detestable to a Rationalist as Foucault and Marx.

Carlyle, Chesterton and Thoreau are about the limit of their philosophical knowledge base.

[samuel]

I don't claim that his work is original (the AI related probably is, but it's just tangentially related to rationalism), but it's clearly presented and is practical.

And, BTW, I could just be ignorant in a lot of these topics, I take no offense in that. Still I think most people can learn something from an unprejudiced reading.

[bnjms]

I think you’re mostly right.

But also that it isn’t what the Yudkowsky is (was?) trying to do with it. I think he’s trying to distill useful tools which increase baseline rationality. Religions have this. It’s what the original philosophers are missing. (At least as taught, happy to hear counter examples)

[ashwinsundar]

I think I'd rather subscribe to an actual religion, than listen to these weird rationalist types of people who seem to have solved the problem that is "everything". At least there is some interesting history to learn about with religion

[bnjms]

I would too if I could but organized religions make me uncomfortable even though I admire parts of them. Similar to my admiration you don’t need to like the rationality types or believe in their program to find one or more of their tools useful.

I’ll also respond to the silent downvoters apparent disagreement. CFAR holds workshops and a summer camp for teaching rationality tools. In HPMoR Harry discusses the way he thinks and why. I read it as more of a way to discuss EY’s views in fiction as much as fiction itself.

[turtletontine]

  For example, I recall being in lot of arguments that are purely "semantical" in nature.

I believe this is what Wittgenstein called “language games”

[throwaway314155]

In spirit of playing said game, I believe you can just use the word "pedantic" these days.

[quickthrowman]

Your time would probably be better spent reading his magnum opus, Harry Potter and the Methods of Rationality.

https://hpmor.com/

[ramon156]

Sounds close to Yuval's book nexus which talks about the history of information gathering

[hiAndrewQuinn]

If you're in it just to figure out the core argument for why artificial intelligence is dangerous, please consider reading the first few chapters of Nick Bostom's Superintelligence instead. You'll get a lot more bang for your buck that way.