Hacker News new | ask | show | jobs
by imh 3440 days ago
You are advocating for something called sequential testing. I'm not going to go into the details, but you don't use the same metrics with sequential testing if you want to control your final error rates. That is, you can't look at the p-value on your t-test and say that's your false positive rate for the sequential test. It's important for reasons you describe, but not exactly idiot proof.