[socketcluster]

Great article. I like the simple point about the hypothetical IQ test sent one week in advance. It makes a strong case about time being the true bottleness. I think this same idea could be applied to most tests.

Implicit in the design of most tests is the idea that a person's ability to quickly solve moderately difficult problems implies a proportional ability to solve very difficult problems if given more time. This is clearly jumping to a conclusion. I doubt there is any credible evidence to support this. My experience tends to suggest the opposite; that more intelligent people need more time to think because their brains have to synthesize more different facts and sources of information. They're doing more work.

We can see it with AI agents as well; they perform better when you give them more time and when they consider the problem from more angles.

It's interesting that we have such bias in our education system because most people would agree that being able to solve new difficult problems is a much more economically valuable skill than being able to quickly solve moderate problems that have already been solved. There is much less economic and social value in solving problems that have already been solved... Yet this is what most tests select for.

It reminds me of the "factory model of schooling." Also there is a George Carlin quote which comes to mind:

"Governments don't want a population capable of critical thinking, they want obedient workers, people just smart enough to run the machines and just dumb enough to passively accept their situation."

I suspect there may be some correlation between High IQ, fast thinking, fast learning and suggestibility (meaning insufficient scrutiny of learned information). What if fast learning comes at the expense of scrutiny? What if fast thinking is tested for as a proxy for fast learning?

What if the tests which our society and economy depend on ultimately select for suggestibility, not intelligence?

[dawnchorus]

>most people would agree that being able to solve new difficult problems is a much more economically valuable skill than being able to quickly solve moderate problems that have already been solved

Do most people agree with that? I agree with that completely, and I have spent a lot of time wishing that most people agreed with that. But my experience is that almost no one agrees with that...ever...in any circumstance.

I don't even think society as a whole agrees with this statement. If you just rank careers according to the ones that have the highest likelihood of making the most money, the most economically valuable tend to be the ones solving medium difficulty problems quickly.

[dasil003]

Really depends on the problem. Cancer, yes. Banking, no.

[weinzierl]

"Implicit in the design of most tests is the idea that a person's ability to quickly solve moderately difficult problems implies a proportional ability to solve very difficult problems if given more time."

I used to share that doubt, especially during my first semesters at university.

However, my experience over the decades has been, that people who solved moderately difficult problems quickly were also the ones that excelled at solving hard and novel problems. So in my (little) experience, there is a justification for that and I'd be definitely interested (and not surprised) to see credible evidence for it.

[exceptione]

I could understand that, as we are inclined to hold this for true: "slow -> not smart".

What we do know is "not smart -> slow", because if you are dumb, you will be (infinitely) slow to answer correctly. But note this might indicate that the first proposition is a common logical fallacy where the modus tollens is applied incorrectly.

The slow but brilliant thinker wouldn't perhaps show up for solving a hard and novel problem, as they might have learned they are stupid, and they might still be slogging trough other problem sets. Other excuses are found in https://almossawi.substack.com/p/slow-and-fast-learners-3-qu...

If you want to test pure ability for deep thought, it will be very difficult to control all variables that affect slow people.

[paulcole]

> I like the simple point about the hypothetical IQ test sent one week in advance.

It’s a simple point but an incorrect one.

If you can work on it for a week, it’s no longer an IQ test. Nobody is saying that the questions on an IQ test are impossible. It’s the fact that there are constraints (time) and that everybody takes the test the same way that makes it an IQ test. Otherwise it’s just a little sheet of kinda tricky puzzles.

Would you be a better basketball player if everyone else had to heave from 3/4 court but you could shoot layups? No, you’d be playing by different rules in an essentially different game. You might have more impressive stats but you wouldn’t be better.

[majes]

> Would you be a better basketball player if everyone else had to heave from 3/4 court but you could shoot layups? No, you’d be playing by different rules in an essentially different game. You might have more impressive stats but you wouldn’t be better.

I think the correct analogy here is that if everyone had to shoot from 3/4 court, you would likely end up with a different set of superstars than the set of superstars you get when dunking is allowed.

In other words, if the IQ test were much much harder, but you had a month to do it, you might find that the set of people who do well is different than who does well on the 1 hour test. Those people may be better suited to pursuing really hard open ended long term problems.

[paulcole]

No, I don’t think that is the correct analogy. The analogy in the blog post is that you (one person) gets a month headstart on the test. You would look like a genius because you’d outscore everyone else who had the time constraint.

Yes, if you play a different game you’ll find different high performers. That is obvious. But it is not what the blog post is saying. It is saying if you let one person play the same game but by different rules, they will look better.

[wavemode]

This is the passage you're citing:

> Consider this: if you get access to an IQ test weeks in advance, you could slowly work through all the problems and memorize the solutions. The test would then score you as a genius. This reveals what IQ tests actually measure. It’s not whether you can solve problems, but how fast you solve them.

You retort that "if you can work on it for a week, then it's no longer an IQ test", but that retort is one that the author would agree with. The author is simply making the argument that, what IQ measures is not necessarily the same kind of intelligence as what is necessary for success in the real world. He's not actually arguing that people should be allowed to take as long as they want on the test, he's simply using that hypothetical to illustrate "what IQ tests actually measure".

[paulcole]

> This reveals what IQ tests actually measure. It’s not whether you can solve problems, but how fast you solve them.

Who is out here arguing that IQ tests only measure whether you can solve puzzles or not?

> You retort that "if you can work on it for a week, then it's no longer an IQ test", but that retort is one that the author would agree with.

Well it would be unreasonable to disagree with because it is less a retort than a simple fact.

[majes]

Yeah, I should clarify - I also don't think the article made the correct analogy. But I more meant that I think the different-game-gets-different-winners-analogy should have been how the article tried to make the point the author ultimately intended.

[halfcat]

Counterpoint to consider: In real life, you can just play a different game. Most people will choose to shoot from 3/4 court instead of running all the way to the other end, because they’re not interested in basketball.

Most people aren’t interested enough to work 100+ hours per week. But we wouldn’t say Elon isn’t better at work ”because he doesn’t even work a 40-hour work week”

It has a lot to do with interest. Michael Jordan isn’t a world class mathematician. Elon isn’t a world class father.

[paulcole]

> Most people will choose to shoot from 3/4 court instead of running all the way to the other end, because they’re not interested in basketball.

I have never once in my life seen anyone do anything close to this. Have you?

[aDyslecticCrow]

> Implicit in the design of most tests is the idea that a person's ability to quickly solve moderately difficult problems implies a proportional ability to solve very difficult problems if given more time. This is clearly jumping to a conclusion. I doubt there is any credible evidence to support this.

I think this approach is effectively testing if a student studied the material. It makes the correlation between memorization and understanding. Recall a piece of information is fast if avaliable.

Its a commonly expressed experience among university students that learning memorization techniques and focusing on solving previous exams is a disproportionately effectively way to pass courses.

It's technically more impressive to pass the exam by never doing a single similar problem before and deriving a solving method or forumla that wasn't memorised.

I took deliberate effort to avoid looking at previous exam question for a course until the week before, since it cased good grades at little value to me long term.

[lo_zamoyski]

> I suspect there may be some correlation between High IQ, fast thinking, fast learning and suggestibility (meaning insufficient scrutiny of learned information). What if fast learning comes at the expense of scrutiny? What if fast thinking is tested for as a proxy for fast learning?

Precisely. Speaking from experience, in school, every claim that I was supposed to accept and reproduce on an exam or in homework was met with a gut response: "Is this really true? Is so, why? How do you know?". I wanted to verify the information and know the justification for believing it, the reason something was true. What's more, I had trouble with the coherence of the claims being made. The physics we are taught in school, for example, raises very serious metaphysical questions. This produced in me a spirit of rebellion. I felt a certain vague disgust for the way things were taught that frustrated my motivation. In some sense, it didn't feel like truth was being treated seriously. The ceremony of education, with all its trappings, is all that was treated seriously. "Getting the grade", not understanding something was what it was all about. It felt like an acrobatics contest and a game of one-upmanship.

Now, sometimes, the justification for a claim was obvious, at least given certain premises (these are often left tacitly assumed, often implied: the danger), but that's not always or perhaps even usually the case. Even in math, a science that can be done from the armchair, we are given formulas and methods that are supposed to be taken on faith and simply used. Through repetition, we are supposed to become better at identifying situations in which we can apply them. But where do these formulas and methods come from? What do they tell us, and how do we know?

And I emphasize "faith": there is no way the valedictorian has verified everything he or she was taught or knows the justification for them. A "good student" keeps up, and since scrutiny and analysis take time and skill - time no student is given especially as the workload piles up, and skill no student possesses - a faithful student, a student who obediently accepts what he or she is being told. You can imagine that blind faith would produce the "perfect student". (Curiously, we are simultaneously commanded to "question everything" - except questioning everything, of course - but then required not to actually practice that advice.)

Now, you could argue that students are too young to understand the justifications for the claims being made, and in practice, we are always relying on faith in some authority. Few people realize how much faith we rely on in our lives. Society entails a certain epistemic deference, even if merely practical or perfunctory. In practice, it is unrealistic not to rely on faith. Faith has its proper place.

Someone might also say that students could be bracketing the information they are receiving. They may simply be entertaining it as a possibility in good faith and playing with it, until verification becomes possible or necessary. Maybe. But given the intellectual immaturity of students, and the obedience at the top, I suspect there is at least a superficial assent given to what they are taught. Otherwise, school is a game to be played, one that, we are told, is an instrument for climbing the ladder of social status. The content doesn't matter. What matters is that you play by the rules of the game and that you play by them well. When you do that, the kingmakers and status granters will throw you a few golden chips and elevate you in the eyes of society. You will be in.

Sounds cynical. After all, wouldn't an institution that wants to select for wisdom also create barriers? Of course, regardless of how effective they are. But the differences cannot be ignored. The intent and purpose are different, for one. The means of selection are another. Education is bureaucratized. We think that bureaucracy will create a "level playing field", eliminating the biases and favoritism that "personal judgement" is bound to entail. But who designs the bureaucracy? What does it actually select for? And does it not often commit the fallacy of confusing features of the method for features of the real?

We're obsessed with rank, and bureaucratic methods make us obsessively so. We imagine there is a sharper slope and a smaller peak than there really is. There is a slope, to be sure. I am no egalitarian. But come on.

Anyway, for all that rambling, what are some of the morals here?

I suppose my first point is that education ought to be focused on first principles first. It ought to be focused on understanding and truth and learning the competence of being able to get there, as that is the whole point. The trivium and quadrivium of old did this. People think of the Middle Ages as some kind period hostile to education. They think it was like the Prussian-style of education (from which modern education gets a lot of its ideas), oriented toward mindless obedience and unquestioned submission to the state. Nothing could be further from the truth. Universities were renowned for open discussion and debate, perhaps most famously in the form of the disputation. The Scholastics were famous for intellectual rigor, a rigor that puts to shame the pompous pretensions of the so-called Enlightenment that never missed the opportunity to erect straw men of the Medievals to ridicule.

Second, rewards and penalties are selection mechanisms. We get the behavior we reward and we get less of the kind we punish. Habits are like this, too: indulging a habit of overeating reinforces and magnifies the habit, while restraint has the opposite effect. What does our education system feed? What does it starve? We should ask this question ceaselessly.

[seec]

My life experience and resulting sentiment is extremely similar to yours.

I feel that the education system is deeply flawed and rewards all the wrong things because we refuse to select based on real factors because of political ideology. I think those that are successful become so despite of it, instead of because of it. When you looks at the biographies or people who truly pushed the enveloppe and changed the world, it becomes evident.

We need to ask if the cost of the education system are really worth the rewards. Considering how large that cost has become nowadays, my premilinary anwser would be no. And I feel that the shift to rent seeking economy as well as reduced innovation and iteration speed is deeply linked to that. Most of the recent growth came from IT, a field that was notorious for be full of dropouts. That should tell you something. Now that the field has been innunduated by college graduate, it has shifted to fully extractive behavior.

Any push back against the system is met with suspition because most people feel they should have a shot at making it big, because they are worth it. In practice, it seems that the inequalities never disappear anyway, and people just have to pay more upfront in order to try to prove themselves. In the long run, it mostly end up exactly as it started and society just pay a dear cost for what is basically unproductive behavior.

You behavior remark is quite on the nose, because from my point of view this is exactly how tyrannies are created. If you get rewarded too much for simply being obedient to the autority in place, overtime any other strategy gets pennalised dispropotionally and you end up with a bunch of sycophant you will never push back against the order, no matter how bad the decisions/rules get.