[nemothekid]

>doing anything economical with the analytics is very rare.

That is a very bold claim and I'm interested if you have any sources to back it up. I'm hard pressed to believe that all of Facebook's design decisions hinge on whatever side of the bed Zuckerberg woke up on.

[anonyfuss]

Belief that numbers can be interpreted usefully without a rigorous statistical approach is an enormous fallacy; I don't really see why the burden of proof lies on me.

Once you do apply rigor, you'll find that what the numbers can tell you provides very little help in guiding application design (unless you're optimizing for extremely simple measurable metrics, eg, Zynga).

Note that Zynga themselves copies a full game design, and then applies metrics to optimizing games for addiction and spendthrift response.

The numbers also can't tell you that what you really need to do is rework your application's entire interaction model to cleanly integrate feature X, Y, and Z -- which will also be far too expensive to even attempt to A/B test.

Most analytics users are simply playing an expensive game of "hot or cold".