Hacker News new | ask | show | jobs
by internet101010 854 days ago
The big tests that matter are always worth the wait. Making sure people don't screw it up by changing things mid-test can be difficult. Replacing the "most likely to be successful" with "most representative of potential rollout group" mindset of stakeholders isn't easy either.

Then you have people that try to get greedy. On more than one occasion I have designed a test where two variables change, results are great, rollout projections are great, the stakeholder attempts to do the rollout without changing the variable that creates incremental expense, and the rollout does not meet projections. Then they reluctantly do what they were supposed to do in the first place and everything is fine.