[jfarmer]

Seems like a straightforward solution would be to get people to opt-in by offering them credits, increased limits, early access to new features, etc.

Universities have IRBs for good reasons.

[Aaargh20318]

A problem with this approach could be that you're now only testing the feature with the kind of people who would sign up for an A/B test. This group may not be representative of your whole user-base.

[jfarmer]

So they'd need more robust experimental designs and statistical methods. They exist.

And unlike the university context, there’s a glut of data.

A basic technique: https://en.wikipedia.org/wiki/Inverse_probability_weighting

Or https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4384809