[calrueb]

Appreciate the insights. It makes sense that this class of metrics have made it into the decision making process, and I am glad it is happening.

The two concerns that come to mind without deeply understanding the problem is that: 1) Measuring a qualitative, nebulous metric like "wellbeing" (which could mean different things for different people) is likely very hard to do right 2) In my experience, things tend to move fast, and experiments often don't run for _that_ long. I would hypothesize that Facebook's negative effects on users is a compounding effect that emerges over the scale of months. Sure, you can leave a small % of users in a holdout group of your experiment, but how often is that getting revisited?

I do like the idea that there are teams out there that are taking it as a goal to positively move these non-engagement metrics. If FB is going to correct course then steps like this are a big part of that.

[mgraczyk]

Yes, wellbeing is a very hard thing to measure. I didn't work on it directly so I can't really weigh in on the high level philosophy, but the general strategy seems to be to measure a lot of things.

As for the holdouts, people do revisit the holdouts extremely often. I'd say Instagram does holdouts better than any other place I'm familiar with (better than most of Facebook). For higher level engineers and product managers (5-6 +), the holdouts are one of the biggest signals for performance review.