[stkni]

>> But if you tweak the hypothesis just a little, the data suddenly confirm it

This is 'data mining' right? And I've occasionally wondered about this, since I don't work in a scientific field but did once make use of the scientific method for some research I did. And yes the findings weren't especially conclusive but I'm not sure I could've tweaked the hypothesis to make it work.

So, had I found something really interesting that didn't fit the hypothesis, is the 'right way' to conduct a new experiment from scratch? So say I did that, and used the 'tweaked' hypothesis, of course I'd find something interesting, because it's already there.

In this new 'pre-registration' framework, how can I correct the problem and pursue the interesting idea but keep the science in-tact? Because, if I used some sort of cross-validation at the outset and I have all the data available I presumably can't change the sample, so the hypothesis presumably has to change.

[probably_wrong]

Refining an experiment is not wrong. What is wrong, like you say, is going on a fishing expedition until you find a result you like.

There are methods to account for follow-up experiments. Bonfaroni correction [1], for instance, requires you to increase your significance level with each new test.

[1] https://en.m.wikipedia.org/wiki/Bonferroni_correction

[soVeryTired]

It's harder than you might think to control for multiple comparisons. The Bonfaroni correction assumes that each experiment is independent, and so penalises correlated experiments unnecessarily harshly.

On the other hand, other tests typically require the researcher to make explicit assumptions on the correlation structure of the experiments despite the fact that it is not directly observable.

[srean]

You are probably thinking of Sidak correction when you state independence is needed. Bonferroni correction does not need independence. You are absolutely right about Bonferroni being a severely conservative correction though -- at least the 'first order' one that uses only the first term of the Bonferroni inequality. One can take more terms to be less conservative but those aren't as easy to apply as you need to know the joint distributions over larger and larger tuples of events.

Another more recent technique for 'exploratory' yet correct technique is to exploit differential privacy and dithering.

[rumcajz]

You can also split the dataset into two parts. Use first part to form a hypothesis. Register it. Then use the second part to confirm/disprove it.

[srean]

>This is 'data mining' right?

That would be datamining done wrong. Its perfectly fine to look at data to provoke new hypothesis. But you should not be using the same data to confirm the hypothesis that it provoked. Either use fresh data or make sure that you still ensure correctness if you are reusing the data.