[cwyers]

The problem with everything you've said is that statistical significance tests are almost always statistical power tests -- do you have enough statistical power given the magnitude of the effect you've seen. The underlying assumption of something like the p-value test is that you are applying to p-value test to all known data sampled from an unknown distribution.

If it is standard laboratory procedure to discard results that are aberrant and to repeat tests, and then to apply the p-value test ONLY to the results that conform to some prior expectation, then the assumptions underlying the p-value test are not being followed -- you're not giving it ALL the data that you collected, only the data that fits with your expectations. Even if this is benign the vast majority of the time -- if 99.9% of the times you get an aberrant result are the result of not performing the experiment correctly -- using the p-value test in a way that does not conform to its assumptions increases the likelihood of invalid studies being published.