[masswerk]

> I can understand why engineers are predisposed to see instantaneous A/B statistics as self-evidently positive

This is the crucial misunderstanding: in actuality, you are running a panel.

(There is no such thing as an A/B test outside of marketing. Running a meaningful panel requires some information on the population, your samples, the homogeneity of those, etc, just to pick the right test, to begin with. Also, you need a controlled setup, which notably includes a predetermined, fixed timeframe for your panel to run. Before this is over, you have no data, at all. You are merely tossing coins…)

[naijaboiler]

Basically what op did was when you got software developers with little understanding of stats or analytics cosplaying as data scientists

[PeterCorless]

Data scientists also do A/B testing on algorithms to see which one has better fit for a use case against real-world, real-time data.

[masswerk]

I mean, if done right – there are dedicated sciences for this –, it's a panel with two samples. To me, the notion of an A/B test is all about tossing the scientific basics over board, in order to get a rough estimate, which we will call good enough. However, there are all those statistical methods, meant to even out the bumps in the road that you will encounter, e.g, if you're running your load-balancer performance A/B test at 2 am versus running it 5 pm, or running it on a Sunday versus running it on a Tuesday.

(In this specific case, as a data analyst, you will probably have an intricate understanding of your population, i.e. your data, the structure of the samples you're running against the algorithms, which you have tailored according to this understanding in the first place. However, while we may assume the best, this may still be what's called pre-scientific knowledge in statistical terms.)