|
|
|
|
|
by placidpanda
1521 days ago
|
|
I am also team bayes for all the reasons you stated, but do want to argue a couple counterpoints: * While you don't have to have a fixed sample size up front, you can still "cheat" in a bayesian analysis if you peek constantly and end early on promising results that you want to win, and let them run longer otherwise. So you want to do something to account for this (put some structure in place, approach with skepticism, laugh and put on sunglasses, whatever). * It's very often useful in practice to have some idea of what kind of answer you're going to see in how long for planning reasons -- for example, rather than your boss saying "I need an answer tomorrow" they say "I need an answer as quick as you can". Bayesian methods give you the flexibility to be risky when you need to and accurately count for uncertainty, but sometimes you still need to predict and strategize around ideas like "We'll be about this certain in 2 days, and about this certain in 1 week, and about this certain in 4 weeks and it seems like planning on next Tuesday is the right call" I've found understanding these frequentist methods to help inform my guesstimates of how experiments will play out with regards to sample size and impact as well as honestly evaluate the trade-offs in evaluating the tests where I wasn't running it -- AB testing is really widespread so I feel like it's important to understand frequentist tests well even if you intend to never use them if you can help it. |
|