[TrainedMonkey]

Blog post is interesting, but is kind of not organized. Almost like author did not spend enough time "deep thinking" about it before doing information dump.

Here is the flow of how to analyze something deeply as I see it:

1. Take your idea to extreme. This is about mapping out overall constraints of the problem. It will help you understand not only the issue you are trying to think about, but also how set of options is constrained by environment of problem lives in.

2. Come up with analogies, understanding constraints and environment will really help you to map out useful analogies.

3. Based on analogies you can come up with real world examples. Because you mapped out constraints and came up with analogies, you will appreciate subtle differences between examples (and the problem you are thinking about).

4. Now, you can check out situations historically similar to your examples to see how those subtle differences affected the outcome.

5. By now your should have sufficient understanding of the problem to set up either thought or real experiments to see how tweaking variables will affect the problem.

All of those connections between steps are in the article, but it is hard to notice them amongst other noise. Really cool thing, that after going through once you can come back to first step with significantly expanded understanding, which will lead to mapping out constraints better, and thus the space whatever you are thinking about lives in.

[tjradcliffe]

Most of these are at least as likely to provide the illusion of understanding as they are to produce understanding.

1) Taking ideas to extremes... which extremes? And are the effects monotonic? We live in a world of many dimensions. I have seen many good ideas shot down by people taking their consequences to extremes along a stupid dimension, or by assuming that the effects of change along a given axis are monotonic. Anti-process arguments tend to go this way, "If we take process to an extreme development will cease, so let's dispense with process!"

2) Analogies are also subject to dimensionality. Everything is similar to everything else in some respects. Arguments by analogy are great ways to mislead you into thinking you understand something you don't. For example, Plato's analogy between individuals and states.

3) Examples are useful, although it is easy to pick bad ones and generalize from them inappropriately (see: any argument about the unique inability of Americans to reduce their gun murder rate to that seen in the rest of the world.) Examples frequently lead to argument by anecdote.

4) Historical analogies are also famously misleading because so many factors change. World War I was not at all like anything that came before it, and decision makers were badly misled by using inappropriate historical examples.

5) This mixes a very bad idea with a very good one: the difference between thought experiments and real experiments is that real experiments tell you about the world, and thought experiments tell you about your imagination. The human imagination is well-known to be a terrible instrument for understanding the world. Imagination is useful for many things. Deep understanding of the world is not one of those things.

If you want to think deeply about a problem you need to test your ideas about it via the discipline of systematic observation, controlled experiment and Bayesian inference. This discipline (not method) is called "science", and it can be applied to anything.

If you really want to get deep into an idea, ask "If this idea were true (or false) what would the consequences be? How would they appear in the world? How could I measure them?"

For example: I have an idea that 500 ml bottles of wine would be a product that had some demand in the market. If this was true you'd expect to see some product offerings in that space (you do, particularly in restaurants, so that increases the posterior plausibility of the idea). There are likely other observations one could make, and test marketing is likely the appropriate experimental approach, although there may well be others.

No amount of imagining is going to give me the information about the way the world is that is required to make this decision, and in general imaginary arguments--arguments based primarily on the contents of an individual's imagination--should be avoided. Philosophers tried to understand the world using the method of imagination for thousands of years, and they failed utterly.

[TrainedMonkey]

1. To the extremes that environment of the problem will be able to support. For example if you want to figure out education one of the extremes would be one teacher per student, other extreme would be the top minds in the field create a single class online and teach it to everyone together (Everyone gets same material, as opposed to different material per student in first case). You immediately see what is different between two, second one enables way to collaboration and idea swapping between students at the cost of face to face to face teaching and individual attention each student receives. In this context you can think of how many students can each of the extremes support.

2. I think analogies are a way of simplifying problem so that you can model it. People usually do it by assigning actors that they have experience with to the models, but that is not necessary. So if we model education as knowledge transfer between actors we can look for other concepts that share similar models.

3. Bad examples are a problem, I have nothing on this one. It might be mitigated by the fact, that on average some people will get good examples and succeed, but that feels like a non-argument.

4. Good point on historical factors, they are still useful to help pinpoint what kind of factors are affecting problem you are thinking about.

5. I think point of thought experiments is to try to come up with an obvious flaw in your experimental setup. Knowing those potential flaws would help you to set up controls in real experiment.

Not everyone is going to succeed in understanding big problems such as education system anywhere close to accurate. Even fewer people will be able to do anything with their understanding. I do not think that is the problem, as long as people try some will make it. I do know for sure that if you do not try, you will definitely never make it.