| There are many more or less obvious ways that people do p-hacking without even realising it. A classic one is looking at eg an eeg topographic plot, notice which areas or channels within an area seem to be more promising, and running stats and follow ups on these. There are of course degrees of these: people may have preregistered which area (let's say prefrontal cortex for example) but leave open which channels (because it is a bit hard to make that exact guesses anyway). There are methods to deal with this (eg cluster permutation analysis) but often people seem to think that they have to choose between averaging between too many channels, thus risking smoothening out and decreasing an existing effect, or cherry-picking channels based on visual inspection of the data, which means artificially increasing an existing effect or even creating an artifactual one. Because people do not actually run a test to pick the channels, they just visually inspect the data, they do not actually realise this is p-hacking. The problem is that determining the researcher's degrees of freedom is not an easy task, and not one that can just be formalised in a p-adjustment technique. There is a huge spectrum of practices around these degrees of freedom, that may happen during any stage of the data processing, that range from obviously to subtly sketchy and problematic. And believe me that often people who do that think that they actually have good practices, and others do p-hacking. Imo the main way to actually avoid this issue is actually being transparent with all the decisions one makes, even if this can reduce the faith on one's results (which actually should be the point of it, if that's the case!). A lot of time shit happens, and often it is hard to predict everything in advance in a preregistration. If the incentive was to just play safe then not much innovation and method experimentation would occur. It is easy to talk about preregistration as panacea in fields with long ago established practices, but much harder when the state of the art wrt both methods and theory may change wildly even in 2 years that may take to run a study. I believe we need better frameworks for rigorous exploratory research. The only paper I have seen to actually take this idea seriously is this one [0], but I believe a lot of research would more honestly fit in such a framework, and not everything should be conceptualised within a hypothesis testing framework. Method-wise, closed testing procedures also seem very interesting for such research (and can work both actually inferentially, but also for extracting hypotheses for further testing), such as [1]. [0] https://pmc.ncbi.nlm.nih.gov/articles/PMC7098547/ [1] https://openpharma.github.io/CTP/articles/closed_testing_pro... |