[bjlorenzen]

As a developer working for a major competitor to airbnb on a shopping page, and having implemented hundreds of experiments on my page, I can say that these guys are way too obsessed with statistical certainty.

Rate of deployment of experiments is a better focus; since all your opponents are bound to copy your winners anyways, you have to rely on the few months edge you've earned before they do so, and constantly maintain that lead.

[thinkmoore]

Unless random high bias means your "edge" is exactly the opposite.

[001sky]

This is about proportionality. Finite window trading strategies need to take into account the link between implementation overhead (time) and overall profitability (linked to time). Just the same way they need to link pricing and profits (linked to pricing). That seems to be the crux of the issue.

[thinkmoore]

Yes, I'll agree with that. Just warning about throwing the baby out with the bath water. Applying quantitative decision making tools is worthwhile, being picky about it can protect us from our emotions and preconceptions.