[jakear]

Why not both? User testing won't help you with questions like "there's a bug in feature X, but the fix will potentially impact feature Y in unforeseen ways. Given we're shipping soon, is it a good idea to apply the fix now or should we apply a much simpler mitigation and postpose the full fix till next iteration so we can get more testing in or even implement a larger refactoring that would resolve this category of bugs by design?"

If feature X is used way less frequently than feature Y, the answer is obvious. If feature Y is used way less frequently than feature X, the answer is obvious. If its a wash, you move on to look at different metrics.

This isn't a hypothetical, I ran this exact scenario yesterday against the VS Code telemetry.

I applied the mitigation, the refactoring is coming next week. :)

[reaperducer]

Why not both?

Because you respect your users and know their privacy has value.

[jakear]

I'm interested in your point of view. If I run a query that tells me X hundred thousand users have used feature Y, have I violated anyone's privacy in your opinion?

Also, would you think it's better to not collect such non personally identifiable user information at the expense of quality and stability of product?

[reaperducer]

If I run a query that tells me X hundred thousand users have used feature Y, have I violated anyone's privacy in your opinion?

If you haven't asked them to specifically opt-in to data collection, then yes. And Microsoft does not ask for permission. There are some programs that do a very good job of asking before they send any crash or problem reports. Microsoft is not one of those companies.

non personally identifiable user information

No such animal.

at the expense of quality and stability of product?

One does not necessarily cause the other.

I'm not in the software field, so I'm sure your experience and mine are different. But I work for a multi-billion dollar healthcare company, and the legal department won't let us touch any sort of telemetry with a ten-foot pole. Somehow we're doing just fine, and have a huge satisfaction rate with our user interfaces, according to the studies that have been done for us.