[omginternets]

What exactly makes you convinced that it works? To be specific: why wouldn’t there be bias in the A/B testing results, too?

There are literally people who give astrological analyses for a living.

[zeroonetwothree]

A/B testing has a ton of issues as well that make it easy to be fooled

https://biggestfish.substack.com/p/data-as-placebo

[akyu]

Of course.

[lIl-IIIl]

We are talking about publication bias, where the decision whether to publish something is biased by the outcome of the experiment.

I think this doesn't really apply to A/B testing, because people are incentivized pay as much attention to negative results as to positive ones.

[zeroonetwothree]

From what I’ve seen there is even more incentive to focus on positive A/B tests. It’s the way you get credit for your work at a company. A negative test is counted as barely anything. So your incentive is to run tons of tests, then cherry pick only the positive ones and announce them widely. Another strategy is to track multiple metrics for each test and not adjust for that when computing p values. But then at the end you only report the one metric that was positive.

[aaaaaaaaaaab]

People are incentivized to pay attention to the result that increases their mid-year bonus the most.

[akyu]

I cannot share the reason I am convinced it works. But I can tell you I am convinced it works.

I'm sure many people here are in similar situations.

[mcswell]

Great minds! I was writing more or less the same thing, you beat me to publication by three minutes.