[betatim]

If you are making a systematic mistake (not necessarily an error, could also be an approximation that doesn't hold) in your analysis, the "5sigma" does not really take that into account.

Struggling to come up with a simple example. Imagine you collect some data, analyse it and see a "3sigma" effect. You decide to collect more data to see if the effect keeps getting bigger or goes away. After collecting a lot more data you get "5sigma". This could be because it is real, or because you have an "unknown unknown"

If you had a systematic (as oppose to a random effect) shift in your analysis, collecting more data will make you more sure that there is an effect. Even though all you are seeing is the effect of the systematic shift.

For almost all analyses in particle physics (or astro or ...) we spend a huge part of our time evaluating "systematics", it isn't uncommon that this part of the analysis takes a lot longer than the nominal result. Unfortunately this only protects you against known unknowns. You can't take into account things you don't think of/check for.

[choffstein]

I suppose if your estimator is biased and not consistent -- due to some sort of omitted variable -- you can end up with "significant" estimates that are completely removed from reality. Great explanation at http://eranraviv.com/blog/bias-vs-consistency/.