[karaterobot]

> The worst outcome, the very worst outcome, is to get a small but non-zero number of sales — say 1 or 2. Because now you're in a bind. Do you double down or pull the plug? Does cold-calling work or not?

That seems like a failure to define success in the first place. That seems like doing an experiment and then deciding how it made you feel. You probably want to declare beforehand what your minimum number of sales would be in order to qualify as a success. I agree that the results can still be ambiguous, but the situation he's describing sounds like a poorly designed experiment anyway.

[wavemode]

> That seems like a failure to define success in the first place.

You are in precise agreement with the article's own text: "So when a startup comes to me with an idea for an experiment, the one thing I tell them is: make sure that there’s a well-defined distinction between success and failure."

[alephnerd]

> You probably want to declare beforehand what your minimum number of sales would be in order to qualify as a success

It is hard to define that and sometimes can be noise or mask important information.

For example, some products have long conversion times due to pricing. This is a critical metric to follow, but would be ignored by the metric you defined.