[mef]

Thanks for this great article.

Both this article and Effective A/B Testing (http://elem.com/~btilly/effective-ab-testing/ slide 51) say changing the proportions mid-test is bad. Can anyone elaborate on why this might be?

For example, if I'm randomly showing my existing landing page or a new landing page to visitors at a 2/3-1/3 split and measuring CTR to the signup page, and I change the proportion to 1/2-1/2, why would that skew results?

[Estragon]

I'm curious about this too. The individual tests should be identifiable regardless of the proportions in which they were assigned.

[tel]

I think it depends somewhat on what the proportions you're talking about are. Ben Tilly's slide seems to be saying that changing already enrolled users from A to B is bad, clearly. I don't think changing the filtering rate on new users would be bad, but I haven't looked at the mechanics in a while.