[agnosticmantis]

An investigator needs to rule out all conceivable ways their modeling can go wrong, among them the possibility of a statistical fluke, which statistical significance is supposed to take care of. So statistical significance may best be thought of as a necessary condition, but is typically is taken to be a sufficient condition for publication. If I see a strange result (p-value < 0.05), could it be because my functional form is incorrect? or because I added/removed some data? Or I failed to include an important variable? These are hard questions and not amenable to algorithmic application and mass production. Typically these questions are ignored, and only the possibility of a statistical fluke is ruled out (which itself depends on the other assumptions being valid).

Dave Freedman's Statistical Models and Shoe Leather is a good read on why such formulaic application of statistical modeling is bound to fail.[0]

[0:https://psychology.okstate.edu/faculty/jgrice/psyc5314/Freed...]