[jrochkind1]

a) 5% is still one in 20. Are you doing 20 trials or more a month? (Of course, it won't be _exactly_ 5% of the time... there's some other statistics we could calculate to say how likely it is to differ by more than some specified delta to 5% depending on how many trials you run... oh my)

b) That's assuming you are using statistics right. It's quite hard to use statistics right.

[kingo55]

That's way better odds than most people's gut feel. Everyone thinks they don't need to run experiments because THEY already know what works.

Did you know, in experimentation programs run by Microsoft, Google and Amazon, roughly two thirds of their ideas have no impact or hurt the metrics they were designed to improve? And yet rookie web Devs or marketing assistants "know" better.

Source: https://www.google.com.au/url?sa=t&source=web&rct=j&url=http... (Read section three of this paper by a Microsoft distinguished engineer)

[nostrademons]

It's actually more like 80% for Google. I spent a lot of time running various experiments on Search.

I'll point out a major difference, though: Microsoft, Google, and Amazon are already highly optimized. They've had millions of man-hours put into optimizing everything from product design to UI to UI element positioning to wording to colors to fonts. When you get a new, beneficial result in a change to Google, it's usually because the environment has changed while you weren't looking, and the user is different.

That doesn't apply to a new business that's just learning how to sell their product. In a new business, by definition, you've spent zero time micro-optimizing your message & product. You can get really big wins almost by accident, if you happen to have stumbled into a product category where users actually want what you're building.