[pessimizer]

> results that speak for themselves.

This is exactly how a midwife explained to me why she uses magic crystals. She told me that there's science, and there's results, and that she's seen the crystals work.

[msrenee]

Obviously they don't work by magical vibration, but are you sure they don't work at all? If the midwife feels and acts more confident from having that tool or the mother feels more relaxed because she thinks they will make the process easier, then the crystals do, in fact, work. They just don't work through the mechanism those individuals think they do.

[rsanek]

I mean, yeah, if she has solid RCT data on thousands to millions of childbirths and has found a statistically significant impact from using the magic crystals, I would support their use. A/B as well as scientific research uses the same basis.

The issue is that in fact the midwife will not have such data. The comparison being made is that A/B testing, if run competently, is pretty close to scientific research, in particular for research related to nudging.

[asdff]

I wonder how many engineers crack open a statistics book to find the correct test versus just plotting box plots and saying "see looks pretty different"

[saalweachter]

To be fair, the more profound a result the less math you need to convince anyone it is the case.

[asdff]

Maybe if you are parroting the result in front of investors instead of statisticians that's the case

[pessimizer]

But if run rigorously, A/B testing is identical to scientific research, and the scientific research fails to show an effect.

[sacrosancty]

The OP was referring to A/B tests that were "perhaps not up to the standard of science", not ones that were already science.