[SketchySeaBeast]

> "Test was successful so it's rolling out to all users, minus a 0.5% holdback population for the next 2 years"

Man, I couldn't imagine being a user in such a situation. "Oh, I guess I'm just not getting the better functionality?" Even worse if I were a paying customer.

[adamesque]

It’s actually usually the paying customers asking via support to be added to the holdback, improved experience or no.

This is more true for larger flags that substantially change the experience and may not implement niche or edge-case functionality. Obviously you want to avoid these kinds of tests if possible but it’s not always possible.

[esafak]

Users should not be allowed to select their treatments; it defeats randomization, which is what allows causal inference.

[mandelbrotwurst]

Sure, they'll be more predictive that way, and simultaneously it's valuable to not piss off your customers.

[esafak]

In that case I would take them out of the experiment and impute the censored data.

https://en.wikipedia.org/wiki/Censoring_(statistics)

[SketchySeaBeast]

Ah, that makes way more sense.

[Brian_K_White]

You are probably the lucky elite who got to keep the functionality you wanted.