[adamesque]

It’s actually usually the paying customers asking via support to be added to the holdback, improved experience or no.

This is more true for larger flags that substantially change the experience and may not implement niche or edge-case functionality. Obviously you want to avoid these kinds of tests if possible but it’s not always possible.

[esafak]

Users should not be allowed to select their treatments; it defeats randomization, which is what allows causal inference.

[mandelbrotwurst]

Sure, they'll be more predictive that way, and simultaneously it's valuable to not piss off your customers.

[esafak]

In that case I would take them out of the experiment and impute the censored data.

https://en.wikipedia.org/wiki/Censoring_(statistics)

[SketchySeaBeast]

Ah, that makes way more sense.