[hn_throwaway_99]

> What he definitely does think is that there have been several in-depth critiques of Langer's work

And one of those in-depth critiques, which is linked to in the post, is by the author of the article himself.

> The critiques are not of the form "Langer is wrong". They are of the form "the experimental design, sample size and statistical analysis do not support the claims Langer is making".

This seems like a distinction that's not really worth making. The author of the post is a statistician, and he's published a detailed critique (see https://news.ycombinator.com/item?id=41974050 ) that says that Langer did the stats wrong. So sure, he is saying "the experimental design, sample size and statistical analysis do not support the claims Langer is making", which seems equivalent to "she is wrong" when you're a statistician.

[PaulDavisThe1st]

Gelman leaves the door reasonably ajar on the possibility that Langer is right about effects in the world, but firmly closes it on the possibility that the statistical analysis Langer presents supports this belief.

[hn_throwaway_99]

Well, we'll just have to reasonably disagree with the final interpretation, then. I will say that from reading this closing section of Gelman's paper, it's about as harsh a condemnation as I've ever seen in an academic paper - he essentially says it's not science that's masquerading as science. Written from one academic to another, that's basically the equivalent of "you're full of shit":

> 4.4. Statistical and conceptual problems go together

> We have focused our inquiry on the Aungle and Langer (2023) paper, which, despite the evident care that went into it, has many problems that we have often seen elsewhere in the human sciences: weak theory, noisy data, a data structure necessitating a complicated statistical analysis that was done wrong, uncontrolled researcher degrees of freedom, lack of preregistration or replication, and an uncritical reliance on a literature that also has all these problems.

> Any one or two of these problems would raise a concern, but we argue that it is no coincidence that they all have happened together in one paper, and, as we noted earlier, this was by no means the only example we could have chosen to illustrate these issues. Weak theory often goes with noisy data: it is hard to know to collect relevant data to test a theory that is not well specified. Such studies often have a scattershot flavor with many different predictors and outcomes being measured in the hope that something will come up, thus yielding difficult data structures requiring complicated analyses with many researcher degrees of freedom. When underlying effects are small and highly variable, direct replications are often unsuccessful, leading to literatures that are full of unreplicated studies that continue to get cited without qualification. This seems to be a particular problem with claims about the potentially beneficial effects of emotional states on physical health outcomes; indeed, one of us found enough material for an entire Ph.D. dissertation on this topic (N. J. L. Brown, 2019).

> Finally, all of this occurs in the context of what we believe is a sincere and highly motivated research program. The work being done in this literature can feel like science: a continual refinement of hypotheses in light of data, theory, and previous knowledge. It is through a combination of statistics (recognizing the biases and uncertainty in estimates in the context of variation and selection effects) and reality checks (including direct replications) that we have learned that this work, which looks and feels so much like science, can be missing some crucial components. This is why we believe there is general value in the effort taken in the present article to look carefully at the details of what went wrong in this one study and in the literature on which it is based.