[masswerk]

I mean, if done right – there are dedicated sciences for this –, it's a panel with two samples. To me, the notion of an A/B test is all about tossing the scientific basics over board, in order to get a rough estimate, which we will call good enough. However, there are all those statistical methods, meant to even out the bumps in the road that you will encounter, e.g, if you're running your load-balancer performance A/B test at 2 am versus running it 5 pm, or running it on a Sunday versus running it on a Tuesday.

(In this specific case, as a data analyst, you will probably have an intricate understanding of your population, i.e. your data, the structure of the samples you're running against the algorithms, which you have tailored according to this understanding in the first place. However, while we may assume the best, this may still be what's called pre-scientific knowledge in statistical terms.)