|
Thanks. My takeaway from the Nintil article is that nobody has performed a good study on the effectiveness of mastery learning vs traditional teaching. All of the studies have some fatal flaw: not randomized, small sample size, study duration too short, interval between exams too long, not providing specialized remedial content to students, or not actually requiring mastery. I think the massive effects shown by software tutoring in the DARPA studies point to the mechanism: frequent exams and specialized remedial content. Good tutoring software continually tests students for mastery, identifies specific misconceptions, and provides specialized remedial content for each misconception. The automated software can perform this iteration for each core concept, multiple times per hour. Students frequently get feedback on problems so they waste little time trying to learn material when they don't have the pre-requisite concepts. Students also frequently pass section mini-exams and enjoy feelings of accomplishment. These positive feelings help with learning. Compare that to the mastery learning studies performed. The studies gave exams once a week or once every 4 weeks. A student with a crucial misconception will struggle for weeks before the they finally understand the content. During that time, they feel frustrated and unmotivated. We need a good study of mastery learning. We also need researchers to design their studies better. IDEA: A new kind of journal with an open study design process. Researchers submit their study proposal, experimental procedures, example raw data, code for cleaning and filtering the raw data, code for statistical analyses, code for generating tables and graphs from data, and a paper template that includes different conclusions based on the values produced by the code. The paper template pulls in the tables and graphs generated by the checked-in code. All of this content is public. Anyone may register an account and provide feedback. Vetted researchers volunteer to review the proposal and code. They receive credit in the resulting paper. When reviewers give LGTM, then the journal and researchers commit to publishing the paper, regardless of the results, and before they have done any experiments. A separate LGTM is required from an experienced statistician. The code includes assertions for sample sizes and valid data ranges. The researchers must record video of themselves as they perform the experiments. They must also record raw data from their instruments. They must upload these recordings and raw data. The reviewers must LGTM the recordings and any PII redactions. The researchers must get LGTM for all changes to the code and paper template. The journal's servers execute the template and generate the final paper. When someone later discovers an error in the analysis or code, they can file a ticket or send a pull-request with a proposed change. The researchers commit to reviewing every issue and PR within a time limit. If they fail to do that, then the reviewers must handle it. If the reviewers also fail to do it, then the journal assigns another qualified volunteer as a new reviewer to handle it. After making a change, the system generates a new version of the paper. Anyone may "star" the paper and receive notifications whenever it changes or there is a change to any of the papers it references. If a paper is withdrawn, the system automatically adds warnings to all papers that reference it. |