[leobakerhytch]

That’s what qualitative data is for. At the very least, talking to customers and reading their feedback. Preferably followed by organizing that data somehow, and using it to better understand the quantitative data you have.

It’s fair comment, though, that undifferentiated ‘engagement’ is rarely a good metric.

[kbenson]

Also, hopefully the people in charge are thinking closely about what time engaged means, for that product. I imagine breaking user actions down to intent, if possible, would help quite a bit with that.

For a product that is supposed to be simpler and save people time compared to the alternative, increasing time using the service might mean people like it and are getting more done so using it more... or it might just mean that it's getting harder to use. Then again, maybe they've graduated from doing simple things with it to complex things, and even though interactions take longer, they're still saving more time and effort overall compared to the alternative and are happy.

I guess the bottom line is I think you have to slice and dice the data a lot of ways and think about what it actually means for your product to be using that data effectively.

[leobakerhytch]

Absolutely. I think that’s why it’s so vital to combine both qualitative and quantitative data.

I doubt there are too many people in tech making this mistake, but without numbers there’s a good chance you fall prey to people’s inaccurate perceptions or explanations for their own behavior.

On the other hand, without those first-hand accounts, it’s all too easy to tell a mistaken story about your numbers. In particular, your users’ sentiment towards their time spent in your app is a guess, unless they tell you.

I think timescale is another crucial dimension to evaluating time-in-app, and whether it’s a positive indicator for your business.

For instance, driving up engagement has doubtless been good for Facebook’s financials in the short- to medium-term. Arguably though, that relentless focus has led to the present political climate where they’re fighting off regulation. Whether that’s an existential threat is yet to be seen (one can only hope), but it certainly casts the metric in a different light.

As you say, it comes down to the fact that there is no silver bullet metric; there’s no substitute for thinking about and dissecting the data you collect.