[hnhg]

Also, doesn't their suggested approach amount to multiple testing? In other words, a kind of p-hacking: https://en.wikipedia.org/wiki/Multiple_comparisons_problem

Edit - and this: http://www.stat.columbia.edu/~gelman/research/unpublished/p_...

[throwaway202351]

Yeah, a good AB testing framework would either refuse to let you break things down too much or have a large warning about the results not being significant, but that doesn't always stop the business-types from trying to wiggle in some way for them to show a win.