[Nextgrid]

I disagree with feedback not being reliable. I think that detailed feedback from someone being not happy give you more details than a heatmap for example. I also think that feedback from a user who takes the time to actually leave feedback (and so is more invested in the product, and likely to give you repeat business) might be more valuable than one-off users.

> I'm also interested in the long term (experienced) users behavior which is not possible with such testing sessions.

Is it not possible to reach out to those users and invite them to such a session in exchange of $$$?

> I don't care about your IP, that does not give me any useful info

True but some malicious third-parties might care, whether it's the analytics service itself (Google Analytics comes to mind) or even a law enforcement request to capture/access such data. You are basically creating a potential liability for the user; some people might not want the software to phone home for certain reasons and I think the default should always be safe so telemetry is "off" by default.

There's also the issue that telemetry is typically opaque and the user has no visibility or control over what is sent, so out of an abundance of caution they opt out. I think a good improvement would be to queue all the telemetry data locally, and then periodically ask the user to review, edit/redact & send it if they want to. Apple has done it relatively well there where if an app crashes they allow you to review the report before sending it, and I actually send these the majority of the time (unless it's a process dealing with sensitive data) despite having OS-level telemetry disabled.

[The_Colonel]

> I think that detailed feedback from someone being not happy give you more details than a heatmap for example.

Detailed feedback is definitely nice, but it's quite rare & not sufficient. It's again one person's view, people also often can't articulate what's wrong. Usage patterns across many users may reveal what's wrong ...

> that feedback from a user who takes the time to actually leave feedback might be more valuable than one-off users.

Both are valuable - one-off users might be people who got confused enough to be discouraged from using the product. That's extremely useful info.

> Is it not possible to reach out to those users and invite them to such a session in exchange of $$$?

Impossible for projects with $0 budget.

It's also very unreliable since people working on artificial test data have very different behavior than when they are working on their production data.

[bobwaycott]

> ... people also often can't articulate what's wrong.

Then you’re not asking the right questions to get useful answers. People often can’t articulate anything well unless they’ve had the practice of doing so. I regularly interact in professional and private settings with people who regularly cannot articulate their thoughts, feelings, or ideas on things. I get them to do so by asking the right questions, digging deeper into what responses they give, and putting it all together.

The questions you ask determine the understanding and clarity you receive.