Hacker News new | ask | show | jobs
by johnmyleswhite 1787 days ago
If you work through the math for CUPED, you'll see that only the aggregate correlation term can be interpreted as being "per-unit" -- all of the other terms in the equation are the same terms you use in your post.

Insofar as I think your intuition is leading you somewhere, I think it's leading you towards a realization that a "diff in diff" approach rather than regression adjustment can increase variance in some settings. But regression adjustment is provably better in essentially all circumstances: the only settings in which it is ever worse than no adjustment are outlined clearly in https://projecteuclid.org/journals/annals-of-applied-statist...

1 comments

Scanning various CUPED related pages, I read that it's a way to reduce the variance, and hence p-value. But CUPED is not changing the lift value (difference [or ratio] in means) between T and C itself (or at least, not in the examples I see). The fallacy I describe computes different means from historic lifts and substracts those. Ie. on the third table, the lift is 4.7%, not -1.6%.