Hacker News new | ask | show | jobs
by 47 3142 days ago
How does this experiment methodology account for cross device mixed experience? Because of the multiple devices use, There will be lot of users that have seen both variation.
1 comments

I think changes over time in device should be like any other unobserved variable like age or country or gender which might affect traffic: averaged out by randomization. The point of randomization is that on average, the two groups will be balanced. (If the intervention itself changes the mix and the changed mix then yields more traffic, then that changes what one might speculate about how it works but the simple result of ads->less-traffic is still true.)