[remram]

What does "tolerating experiments" mean? If they can tell it's an experiment, then isn't your change bad?

Do you mean "tolerate change"? But then you still eventually roll out the change to everyone anyway...

Or do you mean that users would see a different color for the "buy now" button every day?

From a purely statistical point of view, if you select users which "tolerate" your change before you measure how many users "like" your change, you can make up any outcome you want.

[j45]

I suspect it’s because some users will actually be pioneers and early adopters vs believing they are.

This kind of threshold adds some flexibility into the subjectivity of finding the best cohort to test a feature with.

[remram]

Where the best cohort to test with is the one that agrees with you...

You can call this measure "courage" but that is not actually what you are measuring. What you measure is not that different from agreement.

[j45]

I didn’t use the word courage, still I understand what you’re saying.

[remram]

adontz did above, that's what they called this user-tolerance-for-experiments metric. I didn't mean to imply you would too, apologies.

[j45]

Oh, no need to apologize at all.

I could have clarified as well that I was leaning more towards the user-tolerance... or as I like to call it user-guess that this feature might be OK with them :)

Another thing I like about granular and flexible feature flag management is you can really dial in and learn from which features get used by whom, actually.... instead of building things that will collect dust.

[inhumantsar]

I think you might be mixing things up a bit.

the tolerance score wouldn't be tied to a specific change. it's an estimate of how tolerant a person is of changes generally.

it's not that different from asking people if they want to be part of a beta testers group or if they would be open to being surveyed by market researchers.

targeting like that usually doesn't have a significant impact on the results of individual experiments.

[remram]

If only people who like changes like your change, should you really go ahead?

Plus you don't know what that correlates to. Maybe being "tolerant of changes" correlates with being particularly computer-savvy, and you're rolling out changes that are difficult to navigate. Maybe it correlates to people who use your site only for a single task, it would appear they don't mind changes across the platform, but they don't see them. Maybe it correlates with people who hate your site now, and are happy you're changing it (but still hate it).

You can't use a selected subset that is not obviously uncorrelated from your target variable. This is selection bias as a service.