[patio11]

anyone would be able to see that the claimed improvement is well within stddev.

You appear to know words associated with statistics, but treat them as if they were magic incantations utterly disassociated from actual math.

I'll bite: what is the standard deviation in that example. I'm looking for a two part answer: a) a number, b) what the number is measuring.

Let me be intellectually honest with you: there is no basis for assuming that an improvement from 2.0% conversion to 2.6% conversion is not statistically significant.

You need sample sizes to even attempt to do the math. For example, try doing a chi-squared test on 100,000 people converting at 2% versus 5,000 converting at 2.6%.

I'm thinking you'll find that you reject the "just as awful" hypothesis with over 99% certainty.

[videophile]

>what is the standard deviation in that example...You need sample sizes to even attempt to do the math.

First, you're asking the wrong person to supply the missing data. The onus is on the OP to justify their stats, not on the person calling BS on the misuse of statistics.

Given that the original article is missing the critical numbers and presenting the rest in the most self-serving manner possible, the only data anyone has to go by is their own experience with conversion rates. We don't even know what "conversion" means for this website because they don't define it, but that justifies making reasonable assumptions (e.g. users who land on page A end up on page B, for a website in some industry, involving some side-effects to get from A to B that are left undefined by the OP). In my experience, the behavior of users on a web site is quite fickle and dependent on the surrounding advertising campaigns, as well as the weather, the time of day, and the timing of holidays. And maybe a billion other confounding factors. So, is the 5% improvement seen here significant? We don't know, there's no evidence to assume it is, and that's precisely my point.