[kimukasetsu]

This.

Multilevel models relax the assumption of independent observations by specifying that the measures of repeated experimental units are dependent on each other. It's a way of telling your model that it has less information than it would have if all observations came from independent units. Therefore, standard errors of effects are usually larger. Otherwise, they are biased [1].

Since most researches are not aware of multilevel models, they design their experiments and aggregate their data to fit the independence assumption, which is rarely a good idea. Many are not even aware of modeling beyond hypothesis tests, and are unable or unwilling to adjust their analysis for confounding factors or non-sampling errors that arise due to experiment design flaws.

Also, p-values should be deprecated, since a) nil hypothesis are strawmen at best and false by definition at worst [2] and b) they incentivize researchers to not think hard about effect sizes and uncertainty in their problems.

[1] https://academic.oup.com/biomet/article/73/1/13/246001 [2] http://www.stat.columbia.edu/~gelman/research/published/fail...

[nextos]

Many articles in e.g. Nature Genetics are cheating by hacking their p-values. These hacks would be much harder to get away with if, as a starter, they were asked to use hierarchical models and continuous explanatory variables, whenever possible.

The article from Andrew Gelman you cited explains this quite well. In general the review articles and books he has co-authored are incredibly helpful to learn how to avoid common issues that plague statistical inference.

We need to shift away from null hypotheses and p-values towards generative models, model selection and effect sizes. It leads to much more robust inference.