[mjw]

Perhaps you could view this "A/A/B" test as a very crude form of http://en.wikipedia.org/wiki/Bootstrapping_(statistics) method? At least if you're resampling A1 and A2 from a pool A and then doing separate A1/B and A2/B tests and looking at how much the resulting statistic varies between the two runs.

Agreed this is a silly way to go about it, but there better-thought-out bootstrapped confidence tests which could be used if you don't fully trust the distributional assumptions behind (say) the t-test.

[jfarmer]

I wish! The words empirical distribution are music to my ears.

No, usually the rule people use is this: "If A1 and A2 show a statistically significant difference, then do not reject the null hypothesis regardless of A1/B or A2/B."