[embedding-shape]

> Sure, you can spend the weeks to months of expensive and time consuming work it takes to get a fuzzy, half accurate and biased picture of what your users workflows look like through user interviews and surveys. Or you can look at the analytics, which tell you everything you need to know immediately, always up to date, with perfect precision.

Yes, admittedly, the first time you do these things, they're difficult, hard and you have lots to learn. But as you do this more often, build up a knowledge base and learn about your users, you'll gain knowledge and experience you can reuse, and it'll no longer take you weeks or months of investigations to answer "Where should this button go?", you'll base it on what you already know.

[johnfn]

You seem to be interpreting my position as saying that one should only use telemetry to make decisions. Of course, no one reasonable would hold that position! What I’m saying is that only relying on user interviews without supplementing them with analytics would be knowingly introducing a blind spot into how you understand user behavior.

[embedding-shape]

Yes, probably because someone else said "If you dont have analytics you are flying blind" which I initially replied to, then when you replied to my reply, I took that as agreeing with parent, which isn't necessarily true.

> What I’m saying is that only relying on user interviews without supplementing them

I also took your "spend the weeks to months of expensive and time consuming work [...] Or you can look at the analytics" as a "either this or that proposition", where if we're making that choice, I'd go with qualitative data rather than quantitative, regardless of time taken. But probably it comes down to what tradeoffs we're willing to accept.

[johnfn]

Maybe it just comes down to how you interpret "flying blind", because I do tend to agree with that statement. Telemetry is one half of the puzzle, user interviews are the other. Without either I would argue you are flying blind; I think you agree here though.

[zerkten]

In enterprise, you have little chance of getting the real story from end users in many cases. IT will also tell you that things are used one way, only for analytics to tell you it's the opposite. If you spend some of your UX research budget to deep dive on the area you can then finally get to the bottom of it.

I think the root of the complaints here is prioritization. The things they care about are prioritized. Qualitative feedback is likely already telling PMs that something is wrong and really should be fixed, but other feedback has more data supporting it.

[acedTrex]

So if you don't want to spend the time doing that, or as is more accurate in corporate settings, the general turnover of the team is high enough that no one is around long enough to build that deep foundational product knowledge, and to be frank most people do not care enough.

This is why telemetry happens, its faster, easier and more resilient to organizational turmoil.

[embedding-shape]

> This is why telemetry happens, its faster, easier and more resilient to organizational turmoil.

I don't disagree with that, I was mainly talking about trying to deliver an experience that makes sense, is intuitive and as helpful and useful as possible, even in exchange for it taking longer time.

Of course this isn't applicable in every case, sometimes you need different tradeoffs, that's OK too. But that some favor quality over shorter implementation time shouldn't drive people crazy, it's just making different tradeoffs.

[acedTrex]

> even in exchange for it taking longer time.

I think in terms of corporate teams this is the issue a lot of times, people just are not on the team long enough to build that knowledge. Between the constant reorgs, these days layoffs and other churn the no one puts in the years required to gain the implicit knowledge. So orgs reach for the "tenure independent knowledge base.

[hombre_fatal]

Asking users isn't a substitute for usage data.

Usage data is the ground truth.

Soliciting user feedback is invasive, and it's only possible for some questions.

The HN response to this is "too bad" but it's a thought-terminating response.

[Brian_K_White]

The ground truth that I never click on Stargate on Netflix is completely at odds with the actual truth that I love Stargate and want more of it and things like it.

What the ground truth usage data is completely ignorant of is that Netflix's copy is a crappy blurry transfer, and so I got dvds instead.

[dpark]

Telemetry doesn’t tell the “why”. You never clicking in Stargate in Netflix is apparently true, so the telemetry isn’t wrong. It just doesn’t answer why.

[Brian_K_White]

Le duh. The whole point is that the perfectly true data is misleading and uninformative, and the "ground truth usage data" argument has a plot hole.

It's not that it has no value at all, it's just that it's stupid to know one thing (how to collect usage data) and think that is all you need to know and that that obviates all other sources of understanding.

If you wish to collect money from human customers, you have to be some minimum level of human yourself. Talking to your customers is not some icky hardship to be avoided and replaced with nice bash script.

[selcuka]

Sure, but Netflix is not interested in whether you love Stargate or not. Telemetry says that you never click it, so it's ok to remove it from their catalogue (which is correct).

Now, they could've done a better job by increasing the quality, but that's a further (and costly) optimisation.

[mort96]

Netflix should be very interested in a fact like "Netflix has show X, but Netflix subscribers who love show X choose to watch it someplace else due to issues with Netflix".

[Brian_K_White]

It's not correct. I paid someone else for dvds. A little more of that and I may consciously question why I pay both netflix and ebay.

[AlotOfReading]

It goes the other way as well. Usage data isn't equivalent to asking users either. A solid percentage of bad decisions in tech can be traced to someone, somewhere forgetting that distinction and trusting usage data that says it's it's okay to remove <very important feature> because it's infrequently used.

[junon]

This. If I'm forced to use a feature I hate because it's the only way to do something, the "ground truth" reflects that I like that feature. It doesn't tell the whole story.

[groby_b]

Most metrics teams are reasonably competent and are aware of that. Excepting "growth hackers"

I haven't been in a single metrics discussion where we didn't talk about what we're actually measuring, if it reflects what we want to measure, and how to counterbalance metrics sufficiently so we don't build yet another growthhacking disaster.

Doesn't mean that metrics are perfect - they are in fact aggravatingly imprecise - but the ground truth is usually somewhat better than "you clicked it, musta liked it!"

[chillfox]

And yet, the observable evidence of changes in software that collect metrics directly contradict this.

[dpark]

Eh, there are a lot of cases where teams A/B test their way into a product that sucks.

[hombre_fatal]

Yeah, it's not a good discussion without concrete examples.

One: Building a good UX involves guesswork and experiments. You don't know what will be best for most users until you try something. You will often be wrong, and you rarely find the global maximum on the first try.

This applies to major features but also the most trivial UI details like whether users understand that this label can be clicked or that this button exists.

Two: Like all software, you're in a constant battle to avoid encumbering the system with things you don't actually need, like leaving around UI components that people don't use. Yet you don't want to become so terse with the UI that people find it confusing.

Three: I ran a popular cryptocurrency-related service where people constantly complained about there being no 2FA. I built it and polished a UX flow to both hint at the feature and make it easy to set up. A few months later I saw that only a few people enabled it.

Was it broken? No. It just turns out that people didn't really want to use 2FA.

The point being that you can be super wrong about usage patterns even after talking to users.

Finally: It's easy to think about companies we don't like and telemetry that's too snitchy. I don't want Microslop phoning home each app I open.

But if we only focus on the worst cases, we miss out on the more reasonable cases where thoughtful developers collect minimal data in an earnest effort to make the UX better for everyone.

[ragall]

> You don't know what will be best for most users until you try something.

That's because you don't understand your users. If you did, you wouldn't need to spy on them.

> you rarely find the global maximum on the first try

One never finds the "global maximum" with telemetry, at best a local sort-of maximum. To find what's best, you need understanding, which you never get from telemetry. Telemetry tells you what was done, not why or what was in the people's mind when it was done.

[yjftsjthsd-h]

> Asking users isn't a substitute for usage data.

Sure.

> Usage data is the ground truth.

Absolutely not. That's how you get "we buried this feature and nobody used it, so clearly nobody wants it".

[embedding-shape]

> Usage data is the ground truth.

For what, precisely? As far as I know, you can use it to know "how much is X used" but not more than that, and it's not a "ground truth" for anything besides that.

[codedokode]

Then pay for the data if you need it so bad.