[Icy0]

I did! On MESE first, then on Hacker News.

Usually when there's a replication crisis, people talk about perverse incentives and p-hacking. But there's 2 things I want to mention that people don't talk as much about:

- Lack of adequate theoretical underpinnings.

- In the case of math education, we need to watch out for the differences in what researchers mean by "math proficiency." Is it fluency with tasks, or is it ability to make some progress on problems not similar to worked examples?

[JustinSkycak]

> Is it fluency with tasks, or is it ability to make some progress on problems not similar to worked examples?

That's an interesting point. Ideally students would have both. My impression is that the latter is far less trainable, and the best you can do is go through enough worked examples, spread out so that every problem in the space of expected learning is within a reasonably small distance to some worked example.

I.e., you can increase the number of balls (worked examples with problem-solving experiences) in a student's epsilon-cover (knowledge base), but you can't really increase epsilon itself (the student's generalization ability).

But if you know of any research contradicting that, I'd love to hear about it.

> Lack of adequate theoretical underpinnings.

If you have time, would you mind elaborating a bit more on this?

My impression is that general problem-solving training falls into the category of lack of adequate theoretical underpinnings, but I doubt that's what you mean to refer to with this point.

[Icy0]

> That's an interesting point. Ideally students would have both. My impression is that the latter is far less trainable, and the best you can do is go through enough worked examples, spread out so that every problem in the space of expected learning is within a reasonably small distance to some worked example.

I simply mean that researcher team A will claim a positive result for method A because their test tested task fluency, while team B will claim a positive result for method B because their test tested ability to wade through new and confusing territory. (btw, I think "generalization ability" is an unhelpful term here. The flip side to task fluency I think more of as debugging, or turning confusing situations into unconfusing situations.)

> If you have time, would you mind elaborating a bit more on this?

I don't know what good theoretical underpinnings for human learning looks like (I'm not a time traveler), but to make an analogy imagine chemistry before the discovery of the periodic table, specifically how off-the-mark both sides of arguments in chemistry must have been back then.

> My impression is that general problem-solving training falls into the category of lack of adequate theoretical underpinnings, but I doubt that's what you mean to refer to with this point.

By the way, I see problem solving as a goal, not as a theory. If your study measures mathematical knowledge without problem solving, your tests will look like standardized tests given to high school students in the USA. The optimal way to teach a class for those tests will then be in the style of "When good teaching leads to bad results" that Alan Schoenfeld wrote about in regards to NYC geometry teachers.