Hacker News new | ask | show | jobs
by srean 2893 days ago
I haven't looked at the KDD paper, but in general it is the other way round. With sequential hypothesis testing expect to need less data on average.
1 comments

That's highly counter-intuitive to me. Can you try to motivate why that's the case?

My intuition is that you could use any sequential (which I translated to online) technique could be used in a non-sequential context. By that reasoning, there's no way a sequential technique could do better, at best it could be the same.

This is 1940s stuff. Checkout Wald.

Short answer: in sequential testing you can ask at intermediate stages whether a satisfactory confidence has been reached. If yes you are done and if not you can continue. On average you will hit a 'yes' sooner. For non sequential you cannot do this if you care about correctness (). So the sample size needs to be pessimistic for non-sequential protocols and then you are bound to that commitment.

() If your method ensures correctness even after inspection at intermediate stages then its a sequential method by definition. There is some confusion in literature about Bayesian and sequential. They are orthogonal concepts. Both Bayesian and Frequentist test of hypothesis can be sequential

Ah! I get it. Thank you!