[londons_explore]

And without A/B testing, every product you use would be worse. Not only would it be less profitable, but it would also be harder to use, less useful, and less productive.

A/B testing isn't a new thing - I'm sure the inventor of the wheel experimented with different shapes, and the buyer of the hexagonal wheel probably didn't have the best user experience.

Multiply that by the number of people in the world and the number of products people use, and A/B testing is really up there as possibly one of the most beneficial ideas ever.

I really don't understand those who claim it should be banned - I see no way that testing two different versions of a website with people who desire to use that website can bring sufficient harm to outweigh those massive benefits.

[brundolf]

I'm sorry, you could make this case about some kinds of telemetry, but specifically not A/B testing. Speaking from work experience: A/B testing doesn't look into the nuances of usability or productivity, it looks at easy-to-quantify metrics like conversion rates and money spent. These metrics rarely align with a better experience for the user (outside of like, prettier buttons and stuff), and instead tend to result in less-informative, less-agentic software (information and choice often distract from conversions!)

[costcofries]

This is complete BS. I run hundreds of a/b tests each quarter and I specifically refuse to run the types of experiments you allude to. My a/b testing is all about helping users achieve the things (the outcomes) that they want to achieve by using our product in the first place. If we can help them do that, with more ease, then we are creating a better experience.

Perhaps you should just agree that, "not all a/b testing is the same".

[fragmede]

How is that BS? Other companies don't have you there to say no to them and are definitely running the kind of experiments you're too good for.

[costcofries]

Did you even read my whole comment? It's BS because he/she/they blanketed it without taking any nuance which i tried to do with my comment + an example!

Quote - "Speaking from work experience: A/B testing doesn't look into the nuances of usability or productivity, it looks at easy-to-quantify metrics like conversion rates and money spent"

[andrewmutz]

That's not true. I've done A/B tests in business software on how long it takes the user to get their job done on a data-intensive form.

It's a great tool, and its impact all depends on how you use it.

[g_p]

This is an interesting example, and perhaps pushes part of the blame and dislike for A/B testing onto tech companies' incentives.

If you're building a tool to make life easier for the user, something that gives them a better experience is your optimal outcome. This seems like a scenario where A/B can produce a good outcome.

The challenge is when you throw in an ad-based revenue model, and the A/B testing is then optimized for the opposite (eyeball-hours, linear metres scrolled per session, ad spots passed, ads clicked) - engagement-based business models end up (I'd argue) A/B optimizing for the opposite of what their users want, to get them to spend longer doing a task they could have done quicker.

[MikePlacid]

> The challenge is when you throw in an ad-based revenue model

The funny thing is - the ad-based revenue model is not the only possible variant. Last time I’ve checked Facebook’s profits per user were $7 per quarter, that is $28 a year. At the same time I am paying LiveJournal $25 a year for the ad-free version. Just taking my money looks like a much better model in many respects:

- less overhead: a lot of people doing these studies how to force me to look at something I do not want to look at will be free to do something more useful to the society;

- streamlined relationship between me and my publisher: in this model there is no advertiser who can say “I do not like these texts, no revenue for you”.

That’s why I prefer to pay for some Substack authors, like Matt Taibbi and Glen Greenwald, than to try to fish their texts for free amid some sea of “clever” advertising (hey AI testers, I bought this thing already, what’s the point of forcing it on me again and again?).

I kinda wish that Brave model (my money distributed between sites I visited) got more traction. It looks much more healthy.

[sokoloff]

The number of people willing to use Facebook today vastly exceeds the number willing to pay $25/yr to use Facebook.

I bet by at least 4 orders of magnitude and likely 5 or 6.

[Retric]

There are much better methodologies for speeding up worker productivity than A/B testing. A/B testing is designed to extract information from people you can’t do more complicated tests such as eye tracking or motion studies with.

The major issue with A/B testing in the workplace is it causes confusion and slows people down when you change things. Which makes these tests really expensive even if they are seemingly easy to preform. So, I would call it useful but flawed.

[dado3212]

As someone who’s run literally hundreds of A/B tests, many of them on the backs of UX research with users in the field, people have no idea what they want. The anecdata is a place to investigate, but never the end of the journey.

[judge2020]

The fear with direct user research is that, unless you have a team and budget for getting enough of a sample, one-on-ones might not only be unhelpful but actively harmful if you implement something that solves that customers' problem but otherwise gets in the way for other customers.

[brundolf]

I'm having a difficult time imagining a situation where people's actual productivity using a piece of software can be so easily measured. I'm sure it happens, but I think it's safe to say this is the exception to the rule when it comes to A/B testing

[scott_w]

You can measure the time between two key actions that operate as a proxy for task completion.

[andrewmutz]

Data entry

[fragmede]

The specific test that did it for me, is that Facebook ran this experiment where they logged users out and then wouldn't let them log them back in despite the correct password, just to toy with them to see how long/hard they would keep trying to login, in order to see how addicted they were to Facebook.

I was in the "B" group, and felt so humiliated at how many times I tried to reset my password to get into Facebook.

[anonomousename]

Is there any evidence of this? Especially for people that use a password manager, this seems incredibly stupid on Facebooks side.

[rjbwork]

Wow. That is insanely user hostile and borderline gaslighting/psychological torture. That is truly one of the most insane experiments I've ever hard of someone running.

[anthonypasq]

i agree, Zuck should be put on trial for violating the Geneva convention because some people couldnt log into facebook.

[strulovich]

Never heard of this one. Any articles about it?

[fragmede]

It was a footnote in the wake of the main psychological experiments facebook ran on its users back in 2014*, which is overshadowing my searches for this particular detail.

* https://www.forbes.com/sites/kashmirhill/2014/06/28/facebook...

[warkdarrior]

The GP forgot their password, couldn't get back in, and made up a story to mask their shame.

[barrenko]

Others have done it after as well. Hell, it may not have originated at FB.

[andromeduck]

I had this happen to me I think, and another case where half my links/screenshots would randomly get censored. Not just like random blogs/tweets/forums but even to super mainstream stuff like wikipedia/cnn/bbc.

[kuramitropolis]

I still get that sometimes - I think someone left it on for all the Tor nodes in Germany...

[TeMPOraL]

It's a matter of degree and kind.

There's also a neat sleight of hand here. Your inventor of the wheel surely tested multiple variants to optimize for the utility of his invention to the user. The A/B testing that's problematic is about optimizing taking advantage of the user. That doesn't lead to better experience, but the opposite. This is what's increasingly popular, and this is what people complain about or want to see banned.

Related: attention economy is predicated on bad user experience, because it makes money from friction.

[blarghyblarg]

ah yes, the good old "Let's A/B test our capability to influence emotional states using the news feeds."

Totally the same as A/B testing button placement. Totally.

[vippy]

10000% this.

[bumby]

There's a reason Tristan Harris called upon SV to avoid "A/B testing ourselves into the 'gradient descent of mankind'".

My qualms are not so much with the method as the morals that guide it. It's agnostic but when operationalized in a faulty moral framework can definitely lead to bad results.

[xzlzx]

A/B testing crosses a fundamental line when people are the product.

[sciclaw]

Agreed. A/B testing helps you meet a desired goal. The desired goal is where ethical questions come in.

For example, I have used A/B testing to see find ways to help users get a task done with fewer clicks, saving them time.

[r00fus]

> And without A/B testing, every product you use would be worse.

The primary goal of A/B testing is to see what's more profitable.

If that happens to result in better UI that's a side effect.

In fact, it could result in less usability (relevant to this conversation, it probably resulted in the frustrating "algorithm-based" timeline at FB/Twitter/etc).

[milosmns]

> > The primary goal of A/B testing is to see what's more profitable.

Well, I think I'll provide a disagreeing opinion. :)

I assume this opinion probably comes from your past experiences, and I believe it is true in many cases. Since I'm not American and have never worked in an American corporate environment, I can't say what is true over there... but my experience in EU and Canada with A/A/B, A/B/C and typical A/B testing (as well as building such testing tools for others) was not like that.

For example, when building tutorials for users, profitability is far from being the primary objective. Same goes for building documentation, programming languages, open-source software, internal tooling and other such things.

Of course, I get that in the end, profitability is the primary goal of the company (with some exceptions). But I maintain that not all A/B tests have profitability as their primary goal, which makes the previous statement an incorrect generalization IMO.

[SoftTalker]

At least in some cases.

A/B testing lead to the development of effective "dark patterns" in UI that trick users into doing things they don't want or don't understand, and then making it difficult to undo.

[londons_explore]

The hexagonal wheel probably wasn't very profitable either.

[mzd348]

But it comes in handy in the alternate universe where pi=3.

[eecc]

Can you elaborate?

[mzd348]

My thinking is that in this alternate universe where pi=3, circles (with diameter 2 * pi * r) will look like hexagons (which have diameter 2 * 3 * r), so wheels would have to be hexagonal.

[tomrod]

Certainly, because it probably never existed as a function for wheel. Do you have any evidence that the "well duh" criteria wasn't used, and hex wheels show up in the archeological record?

[SahAssar]

> A/B testing isn't a new thing - I'm sure the inventor of the wheel experimented with different shapes, and the buyer of the hexagonal wheel probably didn't have the best user experience.

Consider yourself that inventor, would you A/B test hexagonal vs round?

[andsoitis]

I could have hypotheses around wheel dimensions (e.g. width, diameter), materials, etc. that are absolutely great targets for A/B testing.

[SahAssar]

That sounds like a clear no, you already know the answer to the hypothesized A/B test.

As for the other aspects they sound like great targets for testing within different use-cases, but I'm not sure why that'd be an A/B test as we think of them now.

[AnIdiotOnTheNet]

> And without A/B testing, every product you use would be worse. Not only would it be less profitable, but it would also be harder to use, less useful, and less productive.

Do we live in the same universe? As far as I can tell, software keeps trending worse. Usability is terrible, options and settings keep getting moved around and hidden, software is less responsive than it used to be...

[londons_explore]

You are free to keep using MS-DOS for your computing needs...

But the majority of people choose to use a modern computer, presumably because they find it overall more useful than their old MS DOS computer and software.

Sure - there are gripes, but they must be outweighed by something pretty big for 99.99% of people to choose a new computer over a 30 year old one.

[NaturalPhallacy]

>And without A/B testing, every product you use would be worse.

Worse for whom? I feel like a lot of the A/B testing results in more revenue, a more addictive app, and less user satisfaction, because they're not testing for anything beneficial to the user, because at least with FB, you're not the customer, their advertisers are the customer.

[ouid]

This is so wrong in its conclusion, that its hard to know where to start. First, we should be clear that we are talking about involuntary, undisclosed A/B testing.

I have not experienced a product become better for the user as a result of involuntary A/B testing in my entire adult life.

Producers and consumer have both an adversarial relationship and a mutually beneficial relationship, and the distinction between these two is essentially the split between voluntary A/B tests and involuntary ones. In the adversarial component, the producer is trying to figure out how to extract more money from the consumer, without improving the product. Alternatively, (and equivalently), how to make the product cheaper, but also worse, in a way that yhe customer doesnt notice (with their wallet). A proactive version of the "market for lemons".

For instance, if you A/B test your cancellation process to minimize the number of people who cancel their subscriptions, you will almost certainly do something that makes you some additional money, and is also unambiguously evil.

Any A/B testing that is mutual benefit to consumers and producers can be done with consent, by volunteers. And the miniscule amount of scientific rigor you would lose by doing so is not worth the tremendous sacrifice we have seen in quality of consumables in the past 2 decades (probably longer, but i do not have the personal experience to go longer)

You might be compelled to describe involuntary A/B testing as a strategy for maximizing evil subject to the constraint that it be legal, but it often dips its toes into seeing what is illegal but still profitable, and is capable of fundamentally undermining our legal system and even our political system.

The technology has grown more powerful. The addition of computers that can optimize essentially arbitrary objective functions has serious existential implications for humanity.

A blanket ban on the practice, incurring the total dissolution of any corporate entity found guilty of the practice of involuntary A/B testing, would be a start.

[dlkf]

> I have not experienced a product become better for the user as a result of involuntary A/B testing in my entire adult life.

If you did, how would you know?

[ouid]

well, for starters, there would have to have been a product that improved at all. Those are already rare enough that I can enumerate them, and in each of those instances involuntary A/B testing can be ruled out for other reasons.

When craigslist added the map that shows you where all of the people are offering the thing you are interested in. That was a very good change, but thats pretty far from how craigslist operates.

When dominos stopped serving hot glue on cardboard, its pretty easy to see how that didnt come about by furtive A/B testing. They were pretty confident people would like the new pizza more than the old pizza. So they told them about it. Boy did that work for dominos.

That actually speaks more generally to my point. If you're making a change that you think people will like, you tell them about it, because even if it turns out that they dont like it more, the fact that they thought they would and you did it generates quite a lot of good will for them.

[epolanski]

> And without A/B testing, every product you use would be worse

Imagine thinking that seriously.

[dlkf]

It’s a completely ubiquitous practice. So either it does generally help, or every software company ever doesn’t know what they are doing.

To me the latter view is the one that’s hard to take seriously.

[joehendrey]

UX is orthogonal to the thing most web-based software companies optimise for. Unless users are paying for the service, what the company cares about is user engagement, not user experience. It's not that every software company ever doesn't know what they're doing, it's just that what they're doing is _at best_ tangentially related to improving UX. It is clearly not in the user's best interest that they feel compelled to check Facebook frequently throughout the day, or spend hours scrolling through their feed. You can spin that as the company just improving the experience so that people want to keep using it, but fundamentally it is intentional psychological manipulation.

Within a university, research with human subjects is required to pass an ethical review before it is allowed to proceed. Given the scale and impact of the research conducted by Facebook on its users, it is entirely reasonable to hold them to the same standard.

[dlkf]

> what they're doing is _at best_ tangentially related to improving UX . . . You can spin that as the company just improving the experience so that people want to keep using it, but fundamentally it is intentional psychological manipulation.

So we agree that ab testing is good for optimizing toward a numerical objective. You then seem to think that either:

A) There are simply no numerical objectives that correlate with good ux or

B) Every software company ever is optimizing toward perverse incentives by which they take more money from their users while making their products worse

It’s probably B that you believe, and this is such a myopic and paternalistic view. There are a couple cases where it’s a problem, eg cancellation flows. But this problem is orthogonal to AB testing (try cancelling your newspaper subscription in 1994). AB testing is mostly just trying to improve the rate at which people sign up or buy something, and in this case, your objection hinges on the hidden premise that people are idiots.

> Within a university, research with human subjects is required to pass an ethical review before it is allowed to proceed. Given the scale and impact of the research conducted by Facebook on its users, it is entirely reasonable to hold them to the same standard.

One man’s ponens is another’s tollens. See: https://slatestarcodex.com/2017/08/29/my-irb-nightmare/