[reaperducer]

In my opinion, you're doing it either wrong or lazy.

My company doesn't do telemetry. We do actual user testing with actual users, even to the point of shadowing them as they use the product. The same way it's been done for the last 40 years.

You learn more from interfacing with the wetware than you do grepping logs.

[jakear]

Why not both? User testing won't help you with questions like "there's a bug in feature X, but the fix will potentially impact feature Y in unforeseen ways. Given we're shipping soon, is it a good idea to apply the fix now or should we apply a much simpler mitigation and postpose the full fix till next iteration so we can get more testing in or even implement a larger refactoring that would resolve this category of bugs by design?"

If feature X is used way less frequently than feature Y, the answer is obvious. If feature Y is used way less frequently than feature X, the answer is obvious. If its a wash, you move on to look at different metrics.

This isn't a hypothetical, I ran this exact scenario yesterday against the VS Code telemetry.

I applied the mitigation, the refactoring is coming next week. :)

[reaperducer]

Why not both?

Because you respect your users and know their privacy has value.

[jakear]

I'm interested in your point of view. If I run a query that tells me X hundred thousand users have used feature Y, have I violated anyone's privacy in your opinion?

Also, would you think it's better to not collect such non personally identifiable user information at the expense of quality and stability of product?

[reaperducer]

If I run a query that tells me X hundred thousand users have used feature Y, have I violated anyone's privacy in your opinion?

If you haven't asked them to specifically opt-in to data collection, then yes. And Microsoft does not ask for permission. There are some programs that do a very good job of asking before they send any crash or problem reports. Microsoft is not one of those companies.

non personally identifiable user information

No such animal.

at the expense of quality and stability of product?

One does not necessarily cause the other.

I'm not in the software field, so I'm sure your experience and mine are different. But I work for a multi-billion dollar healthcare company, and the legal department won't let us touch any sort of telemetry with a ten-foot pole. Somehow we're doing just fine, and have a huge satisfaction rate with our user interfaces, according to the studies that have been done for us.

[GordonS]

For some products, yes, user shadowing, interviews and testing is enough.

But in the case of VS Code, I agree with the other commenters that you can benefit from both.

VS Code supports many languages, source control systems, workflows, external systems and extensions, and it's used by a very wide spectrum of users and developers. I can see why they believe telemetry is required to better understand how it's used.

[cookiecaper]

They don't have to be mutually exclusive, do they? You can surely learn some things from large-scale telemetry collection that you can't learn from user studies (e.g., semi-representative "in the wild" performance metrics).

[manigandham]

People act differently when they're being watched. Capturing the normal usage as they work with your product without any oversight can reveal a lot.