[jfkw]

From the article "... To make GitLab better faster, we need more data on how users are using GitLab. SaaS telemetry products, which provide analytics on user behavior inside web-based applications, have come a long way in the past few years. They are an important tool for rapidly improving user experiences because you can understand what users are doing (or not doing) in the app. GitLab has a lot of features, and a lot of users, and it is time that we use telemetry to get the data we need for our product managers to improve the experience."

This doesn't seem like a sufficiently good justification add telemetry features which some users may find objectionable.

Perhaps those resources would be better spent on a feature that let the user who is annoyed by a slow operation to:

- Begin recording user interaction telemetry locally - Perform the slow or buggy operation - Stop recording, generating a data bundle file. - Allow the user to review the data bundle in human-readable format - Optionally take the data bundle to a less secured system as needed - Submit the data bundle to GitLab engineering as a well-filed issue

Done that way, I think most users would welcome the interaction with Gitlab. For telemetry, not so much.

[kazlock]

I'm not arguing for or against tracking, but I disagree with some of these assumptions:

1) 99% of users who have a bad experience are not going to be bothered to figure out how to record, review, and submit their session for the benefit of the service provider. If a page has issues loading, they're just going to give up and move on to the next thing.

2) User tracking isn't really related to performance in the first place. If some server side operation is failing or taking a long time to load, engineers will find out (and probably get paged) with user-agnostic internal performance metrics.

3) User tracking isn't just about finding out what's broken with the site, its about understanding how the site is being used in general so that you can validate your assumptions and make product decisions backed by data.

[codedokode]

> If a page has issues loading, they're just going to give up and move on to the next thing.

It means that they are fine with these issues and don't need to have them fixed.

[kazlock]

Or it means they're just going to switch to $competitor's product that doesn't have this issue.

[TeMPOraL]

Telemetry is a lazy, paternalistic way of improving user experience. It's a method that lets you ignore the actual feedback you're getting from your users, because that involves dealing with actual human beings - who don't always write politely or coherently. Telemetry lets you avoid collecting feedback in a way that respects the user - through in-house studies, by contracting with outside users to perform on-site studies, and just by asking people and reading what they say (and treating it seriously).

GitLab, you're not a cut-throat company run by the Ferengi, desperate to eke out a tiny bit of extra profit by whatever means necessary. You have what it takes to do it right and set an example for others.

[hvidgaard]

That is simply not true. Both matter, but telemetry is something you can trust far more than user feedback. Want to know how much a functionality is used, or how long a user spends doing a particular action - telemetry is the answer.

Compared to direct user feedback where a user might use some functionality twice a year, but rank it high as something that needs to be improved because it's sort of backwards. Instead of improving the performance and flow of the functionality they use hundreds of times daily, because they're reasonably happy with how it works. It's not that you don't want to fix the former, but it's obviously not business critical to perform often, and it's not considered high priority yet.

[TeMPOraL]

> Both matter, but telemetry is something you can trust far more than user feedback.

Disagree. User feedback, however biased or incoherent, is a direct representation of real people with real issues. Telemetry is a stream of context-free data from which you need to infer the hidden user feedback. Telemetry data seems to me to be much more prone to finding in it whatever it is that you want to find, or all the other failure modes that happen when people who analyze it have more skills in statistical tools than in doing science.

> Compared to direct user feedback where a user might use some functionality twice a year, but rank it high as something that needs to be improved because it's sort of backwards. Instead of improving the performance and flow of the functionality they use hundreds of times daily, because they're reasonably happy with how it works.

And all I want, all my life when using software, is for companies to fucking listen to that. If I say I'm reasonably happy with things I spend 95% of my time, then I am reasonably happy. I want you to fix the 5%-time case.

There's no direct relationship between how often you use a feature and how critical it is. To give a simplified example, when I'm using 3D Studio Max, 99% of my time is spent modelling, and only 1% (or less) is spent on saving the file. And yet, without that feature, I wouldn't even buy the program. This is of course a silly example, but most software has plenty of very important features executed very rarely (and some that are executed rarely because the experience sucks). Relying only on naive telemetry analysis is going to lead you to misprioritizing work on improving the product.

[samoa42]

> And all I want, all my life when using software, is for companies to fucking listen to that. If I say I'm reasonably happy with things I spend 95% of my time, then I am reasonably happy. I want you to fix the 5%-time case.

soo much this!

[pilif]

Though you have to be careful not to fall into the https://xkcd.com/1172/ trap.

Sometimes a user might think they need feature A to make task B easier when in reality instead of doing B they actually want to do C but they never thought of it that way.

If you implement A, you take on maintanance burden when C would be way easier to implement and potentially much more useful for the majority of your users.

[TeMPOraL]

The way to tell whether the user needs feature A or C is by talking with them, and watching them use your product in context of their overall work. Telemetry does neither, and will not reveal to you that the user needs C instead of A. On the contrary, it's likely to make you double down on A, because that's what shows up in the data.

[SkyBelow]

>If I say I'm reasonably happy with things I spend 95% of my time, then I am reasonably happy.

Isn't there a decent amount of psychology research showing this isn't the case. Not with happiness with a service in particular, but that we poorly report on our internal state and often are biased by factors that impact our self reporting but not our choices. What people say and what people do do not always align and thus it is possible for there to be a gap between what people want and what people say they want. If you try to meet the latter you can end up losing out to someone who tries to meet the former.

Now, that is research on average people (and particularly average college students), so when dealing with software developers talking about software issues it may not apply.

[afiori]

This seems a very appropriate observations, but the way it applies here is that it implies that feedback needs to be significantly contextualized with how users use your product. A criticism of telemetry would be that there is no reason to believe that you will know more about the meaning of the numbers you are collecting than about the user experience through feedback.

A typical story relevant here is that of a news site boasting about a rise in daily reads, where they define a read with being on a page for 3 seconds... that is clearly a meaningless number that maybe just means users are skimming through all your articles in hope that not all you have is terrible (could happen if you had some good content or a loyal audience)

[TeMPOraL]

There is. But one has to remember Egan's law[0] - it all has to add up to normality. Yes, there's an error term - frequently self-reported motivations and reality don't align. Reasons include people not understanding their motivations well, not being clear about weights of different motivations in them, or refusing to reveal either to researcher[1][2]. But it has to add up to normality. If I say I like your app, then you either you believe me, or assume that I'm wrong - which is not just disrespectful and patronizing in this context, but also either implies that either I in particular am crazy, or it invalidates the very idea of human emotions communicating anything.

And most importantly, even if my self-reporting isn't accurate, you don't know better, and telemetry won't tell you. If I say that I didn't press this button because it was red, you can doubt the accuracy of my report, but you don't get to conclude that my real reason was that the button was square and not round, or that it was on the left and not on the right. Or that the button is useless. Telemetry will only tell you I don't press it. Or perhaps you can run an A/B test to learn that I press the button when it's blue and round and on the right side (but you may miss the fact that I pressed it out of confusion, and reverted its effects three screens down the line).

Software developers are no different than real people, except that they have much more experience with computers - which means both familiarity with the UI and the ability to instinctively form a mental model of what's happening underneath, which sometimes leads us to intuitively avoid doing things that would break the application :).

--

[0] - https://wiki.lesswrong.com/wiki/Egan%27s_law

[1] - Plenty of motivations could be embarrassing, or weird. E.g. for a long time, I avoided using GNOME desktop and related software, simply because I had a visceral revulsion towards their logo, which featured a footprint. It's a kind of motivation that did affect me, but I wouldn't be too willing to reveal unless asked directly.

[2] - Also people sometimes report on motivations behind what they want to do, but not on the incentive conflict they're facing, and they then proceed to do something you wouldn't expect. This leads to silly arguments like "something is good because of revealed preferences", which latch on the difference between expressed preferences and observed behavior, completely ignornig the fact that observed behavior is heavily incentivized, essentially making people go against their preferences.

[andrekandre]

> Relying only on naive telemetry analysis is going to lead you to misprioritizing work on improving the product.

yes, but it does let you know where to place the ads and “special offers”

[TeMPOraL]

Which matters primarily if you're a Ferengi.

[hvidgaard]

> Relying only on naive telemetry analysis is going to lead you to misprioritizing work on improving the product.

Which is exactly why I said both matters.

[NullPrefix]

Frequency of usage and importance may not necessarily correlate.

You can drive far by only using gas pedal and take the foot off it to decelerate. Does it mean that brake pedal is not important, because usage telemetry lists it as "rarely used, user does not know what he wants" ?

[beojan]

You've just stated exactly why you shouldn't trust telemetry. You improve the parts that are good enough already and not the parts that are so bad users avoid them.

[hvidgaard]

I didn't want to write an essay, so let me give you an example instead.

We've had a customers insisting that a particular feature was very important for them, however telemetry showed that they used that particular feature last time more than 26 months ago, and they discarded the resulting document anyway. Drilling down revealed that everyone thought that everybody else was using it sometimes, while the one user that had a need for it actually had a different and far more efficient workflow.

My point is that my department have telemetry, the support team and PO have user feedback. If those two versions of the truth does not match up, we discuss it, often involving some users. Neither is perfect, and if you have to choose, obviously listening to the users seems preferable. Just remember that if Henry Ford build what people asked for, he would have made faster horses (quote is probably not even from him, but it gets the point across).

To improve and make innovation you need many things, but especially empirical (telemetry), observational (watch the users), and anecdotal (listen to the users) evidence. Only with that you can make an informed decision.

[samoa42]

> Telemetry is a lazy, paternalistic way of improving user experience

> That is simply not true... but telemetry is something you can trust far more than user feedback

nutin said

[rkagerer]

You don't need telemetry or telepathy to hear the chorus of users screaming at the top of their lungs where to shove your telemetry feature.

[jkdufair]

That sounds like recipe for near-zero participation. I think both the intent and execution are spot-on here for GitLab. They are offering a very straightforward option for using GitLab, telemetry-free.

[emilycook]

Hello! Thanks for the feedback, we opened an issue here that might help clear some of this up: https://gitlab.com/gitlab-org/growth/product/issues/164