[hombre_fatal]

Asking users isn't a substitute for usage data.

Usage data is the ground truth.

Soliciting user feedback is invasive, and it's only possible for some questions.

The HN response to this is "too bad" but it's a thought-terminating response.

[Brian_K_White]

The ground truth that I never click on Stargate on Netflix is completely at odds with the actual truth that I love Stargate and want more of it and things like it.

What the ground truth usage data is completely ignorant of is that Netflix's copy is a crappy blurry transfer, and so I got dvds instead.

[dpark]

Telemetry doesn’t tell the “why”. You never clicking in Stargate in Netflix is apparently true, so the telemetry isn’t wrong. It just doesn’t answer why.

[Brian_K_White]

Le duh. The whole point is that the perfectly true data is misleading and uninformative, and the "ground truth usage data" argument has a plot hole.

It's not that it has no value at all, it's just that it's stupid to know one thing (how to collect usage data) and think that is all you need to know and that that obviates all other sources of understanding.

If you wish to collect money from human customers, you have to be some minimum level of human yourself. Talking to your customers is not some icky hardship to be avoided and replaced with nice bash script.

[selcuka]

Sure, but Netflix is not interested in whether you love Stargate or not. Telemetry says that you never click it, so it's ok to remove it from their catalogue (which is correct).

Now, they could've done a better job by increasing the quality, but that's a further (and costly) optimisation.

[mort96]

Netflix should be very interested in a fact like "Netflix has show X, but Netflix subscribers who love show X choose to watch it someplace else due to issues with Netflix".

[Brian_K_White]

It's not correct. I paid someone else for dvds. A little more of that and I may consciously question why I pay both netflix and ebay.

[AlotOfReading]

It goes the other way as well. Usage data isn't equivalent to asking users either. A solid percentage of bad decisions in tech can be traced to someone, somewhere forgetting that distinction and trusting usage data that says it's it's okay to remove <very important feature> because it's infrequently used.

[junon]

This. If I'm forced to use a feature I hate because it's the only way to do something, the "ground truth" reflects that I like that feature. It doesn't tell the whole story.

[groby_b]

Most metrics teams are reasonably competent and are aware of that. Excepting "growth hackers"

I haven't been in a single metrics discussion where we didn't talk about what we're actually measuring, if it reflects what we want to measure, and how to counterbalance metrics sufficiently so we don't build yet another growthhacking disaster.

Doesn't mean that metrics are perfect - they are in fact aggravatingly imprecise - but the ground truth is usually somewhat better than "you clicked it, musta liked it!"

[chillfox]

And yet, the observable evidence of changes in software that collect metrics directly contradict this.

[dpark]

Eh, there are a lot of cases where teams A/B test their way into a product that sucks.

[hombre_fatal]

Yeah, it's not a good discussion without concrete examples.

One: Building a good UX involves guesswork and experiments. You don't know what will be best for most users until you try something. You will often be wrong, and you rarely find the global maximum on the first try.

This applies to major features but also the most trivial UI details like whether users understand that this label can be clicked or that this button exists.

Two: Like all software, you're in a constant battle to avoid encumbering the system with things you don't actually need, like leaving around UI components that people don't use. Yet you don't want to become so terse with the UI that people find it confusing.

Three: I ran a popular cryptocurrency-related service where people constantly complained about there being no 2FA. I built it and polished a UX flow to both hint at the feature and make it easy to set up. A few months later I saw that only a few people enabled it.

Was it broken? No. It just turns out that people didn't really want to use 2FA.

The point being that you can be super wrong about usage patterns even after talking to users.

Finally: It's easy to think about companies we don't like and telemetry that's too snitchy. I don't want Microslop phoning home each app I open.

But if we only focus on the worst cases, we miss out on the more reasonable cases where thoughtful developers collect minimal data in an earnest effort to make the UX better for everyone.

[ragall]

> You don't know what will be best for most users until you try something.

That's because you don't understand your users. If you did, you wouldn't need to spy on them.

> you rarely find the global maximum on the first try

One never finds the "global maximum" with telemetry, at best a local sort-of maximum. To find what's best, you need understanding, which you never get from telemetry. Telemetry tells you what was done, not why or what was in the people's mind when it was done.

[yjftsjthsd-h]

> Asking users isn't a substitute for usage data.

Sure.

> Usage data is the ground truth.

Absolutely not. That's how you get "we buried this feature and nobody used it, so clearly nobody wants it".

[embedding-shape]

> Usage data is the ground truth.

For what, precisely? As far as I know, you can use it to know "how much is X used" but not more than that, and it's not a "ground truth" for anything besides that.

[codedokode]

Then pay for the data if you need it so bad.