[napworth]

In most cases, A/B testing will actually do negative damage than if you didn't test it at all.

[jpster]

Depends on if the tester actually knows what they're doing. https://blog.kissmetrics.com/your-ab-tests-are-illusory/

[ssharp]

In most cases, flying a plane yourself will result in more negative damage than having a trained pilot fly you.

[ssharp]

Interested in further thought here.

[jdpedrie]

[citation needed]

[eddigo]

And I would bet in a large subset of those cases it's because the person implementing those tests and the person interpreting the results aren't the same. And that neither understands basic statistical hypothesis testing.