[withinboredom]

Interesting. At one place I worked, employees were excluded from experiments (they had to enable the flag personally to see them) by default. At one point, we had so many experiments that literally nobody except employees were using "that version" of the software. Everyone else was using some slightly different version (if you counted each feature as a version), and there were thousands and thousands of versions in total.

We ended up creating just ~100 versions of our app (~100 experiment buckets), and then you could join a bucket. Teams could even reserve sets of buckets for exclusive experimentation purposes. We also ended up reserving a set of buckets that always got the control group.

You've approached it a different way, and probably a more sustainable way. It's interesting. How do you deal with the bias from your 'more courageous' people?

[adontz]

>> How do you deal with the bias from your 'more courageous' people?

That's a great question. We had no general solution for that. We tried to survey people, but results were inconclusive, not statistically significant.

[withinboredom]

I mean that "courageous" people are more likely to take risks and accept new features and thus probably more likely to be attracted to novelty (see: Novelty Effect) and require longer experiments to understand the actual impact.

[tauntz]

> At one point, we had so many experiments that literally nobody except employees were using "that version" of the software. Everyone else was using some slightly different version

Was this at Spotify by any chance? :)

[withinboredom]

No.