[jjk166]

The problem here comes from poor experiment design. Seeing how many sales you get from 100 cold calls has no control, and produces a single datapoint. No matter what number you get, be it 0, 2, or 37, will be useless for predicting future sales.

First, you need a sample size that will produce a statistically significant result. Cold calls are expected to have a low success rate and be highly variable. 2 in 100 could very well be a massive success. 0 in 100 wouldn't rule out its viability. If instead you had a sample size of 2000, then you'd get a very good signal to noise ratio.

Then, you need controls. Look up statistical design of experiments (DOE), you'll find efficient ways for finding how much various different factors affect your results. Basically split up those 2000 calls into groups and vary things as you go, so instead of testing whether a specific cold call technique is working, you can see if cold calls in general work.

Finally, understand what success means. A business does not need to rigorously prove that it's methods are optimal, it needs only to find an adequate strategy to achieve its objectives. You should know what it costs to make 2000 cold calls, you should know how much revenue is generated from a sale, there should be a specific threshold where the number of sales justifies the cold calls. You should be doing a process capability study that you are sufficiently over this threshold that, given expected variation, you will remain above it most of the time. What is your Cpk? How much does it vary among the different sub-experiments you performed?

At the end of the day, you're still going to need to make decisions with incomplete information, but don't pretend that you're making a data driven decision when you're not. The worst thing is not an inconclusive experiment, the worst thing is an experiment with an erroneous result you mistake for conclusive.

[brabel]

> If instead you had a sample size of 2000, then you'd get a very good signal to noise ratio.

In modern statistics, even a small number of samples can be considered enough to get a satisfactorily small error range, as long as the sample is random and representative of the population. I would think 2,000 samples is far more than strictly required if you're able to sample from your target market.

[vharuck]

It's true that people often incorrectly dismiss results with sample sizes below 100. But, for rare events, you really do need large samples. Otherwise, you'll only ever be able to confidently identify massive differences.

[jjk166]

Average results for cold calling are a 4.8% success rate, meaning with 2000 cold calls you'd expect less than 100 hits for a good campaign. This in turn is highly variable with field, a 1% success rate might be high, especially for a product with no pre-existing market presence. And it's not enough to see whether you're above a certain number, you need to know the variance to predict future performance. Maybe the magic number is 1500 or something, the number will no doubt vary based on the peculiarities of the experiment, including the product and the company, but to see a signal with 2 standard deviations of significance, you need a sample size that you would expect to produce about 20 times the expected noise level.