[stdbrouw]

Daniel Kahneman (from his autobiography, but he tells the same story in Thinking, Fast And Slow):

> I had the most satisfying Eureka experience of my career while attempting to teach flight instructors that praise is more effective than punishment for promoting skill-learning. When I had finished my enthusiastic speech, one of the most seasoned instructors in the audience raised his hand and made his own short speech, which began by conceding that positive reinforcement might be good for the birds, but went on to deny that it was optimal for flight cadets. He said, “On many occasions I have praised flight cadets for clean execution of some aerobatic maneuver, and in general when they try it again, they do worse. On the other hand, I have often screamed at cadets for bad execution, and in general they do better the next time. So please don’t tell us that reinforcement works and punishment does not, because the opposite is the case.” This was a joyous moment, in which I understood an important truth about the world: because we tend to reward others when they do well and punish them when they do badly, and because there is regression to the mean, it is part of the human condition that we are statistically punished for rewarding others and rewarded for punishing them. I immediately arranged a demonstration in which each participant tossed two coins at a target behind his back, without any feedback. We measured the distances from the target and could see that those who had done best the first time had mostly deteriorated on their second try, and vice versa. But I knew that this demonstration would not undo the effects of lifelong exposure to a perverse contingency.

See https://en.wikipedia.org/wiki/Regression_toward_the_mean

[davidamarquis]

First, the demonstration Kahneman arranged is simply a demonstration of regression to the mean and is not evidence of the validity of his hypothesis.

Is the hypothesis itself wrong? We can test the validity of the hypothesis on all people by testing it on a subset: teachers with expertise in the subject they are teaching. Kahneman hypothesis implies that such teachers reward students only when their level of achievement is higher than it has ever been in the past and punish them whenever their achievement is below their highest level of achievement.

This does not hold true in my experience of my own teachers. I was occasionally praised for doing well or punished (in some sense) for doing badly but much more often when I spoke with my teachers they remembered trends: I started bad and got better slowly, I started off well but seemed to get lazy, etc.

In my experience teaching first year math courses I would only automatically remember individual achievements if they were very surprising. For example, a student who was failing suddenly moving into the top of the class on a test.

I think people have a natural tendency to focus on these kind of outlying events and assign them special significance that they may not actually have. Although Kahneman does not mention this tendency his hypothesis suggests he also believes this.

But compared to the interesting outlying events the vast majority of student's achievements are not memorable. This means a teacher will not automatically remember them. If the teacher does not choose to remember them then they will be forgotten.

Since a teacher's capacity to remember is limited, trying to remember all student's achievements or failures was low on my priority list. Knowing a student's general trajectory lets you tailor your approach when working with them on a problem whereas individual achievements or failures are usually just noise. Other teachers I knew seemed to feel the same way.

Of course my own experience is only a data point against Kaneman's hypothesis about people in general. I am sure there could be types of teaching environments where something like what he is suggesting could be true. If the performance of the students on a particular test were tied to the teacher's compensation or the opinion of people they respect or who have power over them then I can see how a student's performance on that test would get highest priority in a teacher's mind and this could lead to reactions of the kind Kaneman describes.

[stdbrouw]

> Kahneman hypothesis implies that such teachers reward students only when their level of achievement is higher than it has ever been in the past and punish them whenever their achievement is below their highest level of achievement.

Not really, Kahneman is talking about doing better or worse than expected, not about your best or worst performance.

Regression to the mean arises when following up on any deviation from the mean – though of course the more extreme the deviation, the more pronounced the effect.

Similarly, the effect of regression to the mean is smaller when measuring longer periods of time (less measurement error means fewer fluke outcomes) but it doesn't disappear.

Of course, it's perfectly possible that strict teachers are truly beneficial even when accounting for regression to the mean, it doesn't have to be one or the other, and Kahneman certainly doesn't prove anything of the sort, but it does shift back the burden of proof to those who would claim that strict or even borderline abusive teaching is helpful.

[davidamarquis]

> Not really, Kahneman is talking about doing better or worse than expected, not about your best or worst performance.

At a given point in time there are two quantities we need to worry about: how the teacher expects the student to perform at that point in time and the student's ability to perform at that point in time.

If the teacher is rational and has seen the student perform many times these quantities will be the same. I think the outcome is probably optimized if these quantities are the same at all times but in the real world there may large disparities in the values which can fluctuate over time.

Nevertheless, suppose it were the case that the values are always equal. Then Kahneman's hypothesis is obviously wrong. The student will perform better than expected roughly the same number of times they perform worse than expected (in the long run, obviously).

Regression to the mean exists, yes, but has no impact on the distribution of the events of being above the mean or below the mean.

I supposed that Kahneman would recognize this fact and he his reference to regression to the mean was based on a more complicated but much more realistic model where the expectation is not always aligned with reality. In such a model there must be a mechanism for relating the two quantities after each new performance/test/review.

I tried to think of how this model would work and suggested Kahneman meant that expectations were adjusted based on previous best values. I should have been clearer that other mechanisms were possible and that I was only guessing which Kahneman meant. Then I argued that based on my experience this is a poor model of how teachers punish/reward their students in general because it is unnatural and unhelpful to the task at hand. As this model is not valid his conclusion that people are incentivized to punish each other is specious.