| Completely unsurprised. Experimentation-done-right is too expensive and too ambiguous to sell as a product. Every product experiment requires a complex set up, a lengthy running period across a huge base of users, and then heavy analysis in order to achieve statistical confidence over a specific feature's impact on a business metric. That "confidence" is often represented by a subpercentage point that may or may not be statistically significant. Fun problem for the data scientist, plain hell for the PM. In my company which uses experimentation for everything, each A/B test requires two weeks before the Product Manager can even see the results. Two weeks of waiting for a confusing, contradictory dashboard that can't be taken at face value, that needs careful, human analysis before it can be called a "win". That slowness is fine for high-traffic, high-risk & high-value lines of business. But it's not fine when you're releasing feature that aren't just optimizations. Competitors like LaunchDarkly and Split.io have recognized that critical difference, I think. They know that causality is expensive, and are particularly aware that the fine line between feature release and metrics impact is tied too heavily with a company's politics, i.e. it chafes against the intuition of executives. Instead, they offer experimentation as an add-on to their developer tools. You can experiment if you need to, but it doesn't obligate you to do so. That goal is much more realistic than the Optimizely's current goal: "helping our customers win in a digital-first world". |
But then they are asked to wait! Wait for the concept to mature within the org. Time being of critical importance most orgs end up getting frustrated as what they initially started with doesn't hold true now. And when you already have lots of money, it's easy to switch.