[emckay]

Low participation rate shouldn't matter too much for an RCT right? Just makes the sample smaller so finding statistically significant results is harder.

Different levels of Montessori authenticity make the results even more impressive. They do have some inclusion criteria, like 2/3 of the teachers must be AMI/AMS certified but even so I'd expect a lot of these public school montessori programs to be less "true montessori" than what you'd get at a fully certified AMI/AMS school.

[cool_dude85]

I think the risk is that there is some systematic difference between those who chose to participate and the overall population of public Montessori kids. For instance, maybe those with high incomes disproportionately chose to participate, and Montessori strengthens learning for this group, but if we could measure the whole population the result is more mixed. It can't be a fully RCT if there's some kind of opt-in provision (which is not to say that an opt-in provision is bad, or a study that is not fully RCT is irrelevant).

[mmooss]

This is a speculative criticism about a hypothetical problem. How random was the study?

[notahacker]

https://www.pnas.org/doi/10.1073/pnas.2506130122 You could have answered your own question by reading the abstract, which makes it clear that the OP's conjecture was correct: the lotteries were random or somewhat random, but the groups which consented to the study were notably different, with the treatment group being richer, more educated and whiter. They did of course attempt to control for this, whether the controls were adequate or omitted other underlying differences is another question.

[mmooss]

> You could have answered your own question by ...

Who cares? It's not about me or someone else (or you), it's about the issues at hand. If the commenter wants to make a claim, they are welcome to.

People on HN can't read a study without finding one of the few methodological flaws they are aware of - as if that's some form of serious analysis.