[brundolf]

I'm sorry, you could make this case about some kinds of telemetry, but specifically not A/B testing. Speaking from work experience: A/B testing doesn't look into the nuances of usability or productivity, it looks at easy-to-quantify metrics like conversion rates and money spent. These metrics rarely align with a better experience for the user (outside of like, prettier buttons and stuff), and instead tend to result in less-informative, less-agentic software (information and choice often distract from conversions!)

[costcofries]

This is complete BS. I run hundreds of a/b tests each quarter and I specifically refuse to run the types of experiments you allude to. My a/b testing is all about helping users achieve the things (the outcomes) that they want to achieve by using our product in the first place. If we can help them do that, with more ease, then we are creating a better experience.

Perhaps you should just agree that, "not all a/b testing is the same".

[fragmede]

How is that BS? Other companies don't have you there to say no to them and are definitely running the kind of experiments you're too good for.

[costcofries]

Did you even read my whole comment? It's BS because he/she/they blanketed it without taking any nuance which i tried to do with my comment + an example!

Quote - "Speaking from work experience: A/B testing doesn't look into the nuances of usability or productivity, it looks at easy-to-quantify metrics like conversion rates and money spent"

[andrewmutz]

That's not true. I've done A/B tests in business software on how long it takes the user to get their job done on a data-intensive form.

It's a great tool, and its impact all depends on how you use it.

[g_p]

This is an interesting example, and perhaps pushes part of the blame and dislike for A/B testing onto tech companies' incentives.

If you're building a tool to make life easier for the user, something that gives them a better experience is your optimal outcome. This seems like a scenario where A/B can produce a good outcome.

The challenge is when you throw in an ad-based revenue model, and the A/B testing is then optimized for the opposite (eyeball-hours, linear metres scrolled per session, ad spots passed, ads clicked) - engagement-based business models end up (I'd argue) A/B optimizing for the opposite of what their users want, to get them to spend longer doing a task they could have done quicker.

[MikePlacid]

> The challenge is when you throw in an ad-based revenue model

The funny thing is - the ad-based revenue model is not the only possible variant. Last time I’ve checked Facebook’s profits per user were $7 per quarter, that is $28 a year. At the same time I am paying LiveJournal $25 a year for the ad-free version. Just taking my money looks like a much better model in many respects:

- less overhead: a lot of people doing these studies how to force me to look at something I do not want to look at will be free to do something more useful to the society;

- streamlined relationship between me and my publisher: in this model there is no advertiser who can say “I do not like these texts, no revenue for you”.

That’s why I prefer to pay for some Substack authors, like Matt Taibbi and Glen Greenwald, than to try to fish their texts for free amid some sea of “clever” advertising (hey AI testers, I bought this thing already, what’s the point of forcing it on me again and again?).

I kinda wish that Brave model (my money distributed between sites I visited) got more traction. It looks much more healthy.

[sokoloff]

The number of people willing to use Facebook today vastly exceeds the number willing to pay $25/yr to use Facebook.

I bet by at least 4 orders of magnitude and likely 5 or 6.

[Retric]

There are much better methodologies for speeding up worker productivity than A/B testing. A/B testing is designed to extract information from people you can’t do more complicated tests such as eye tracking or motion studies with.

The major issue with A/B testing in the workplace is it causes confusion and slows people down when you change things. Which makes these tests really expensive even if they are seemingly easy to preform. So, I would call it useful but flawed.

[dado3212]

As someone who’s run literally hundreds of A/B tests, many of them on the backs of UX research with users in the field, people have no idea what they want. The anecdata is a place to investigate, but never the end of the journey.

[judge2020]

The fear with direct user research is that, unless you have a team and budget for getting enough of a sample, one-on-ones might not only be unhelpful but actively harmful if you implement something that solves that customers' problem but otherwise gets in the way for other customers.

[brundolf]

I'm having a difficult time imagining a situation where people's actual productivity using a piece of software can be so easily measured. I'm sure it happens, but I think it's safe to say this is the exception to the rule when it comes to A/B testing

[scott_w]

You can measure the time between two key actions that operate as a proxy for task completion.

[andrewmutz]

Data entry