Not saying it makes it okay, but every company in existence does this to some extent. Everyone does A/B testing with the intent to alter user behavior (aka psychology) to increase their profits.
And without A/B testing, every product you use would be worse. Not only would it be less profitable, but it would also be harder to use, less useful, and less productive.
A/B testing isn't a new thing - I'm sure the inventor of the wheel experimented with different shapes, and the buyer of the hexagonal wheel probably didn't have the best user experience.
Multiply that by the number of people in the world and the number of products people use, and A/B testing is really up there as possibly one of the most beneficial ideas ever.
I really don't understand those who claim it should be banned - I see no way that testing two different versions of a website with people who desire to use that website can bring sufficient harm to outweigh those massive benefits.
I'm sorry, you could make this case about some kinds of telemetry, but specifically not A/B testing. Speaking from work experience: A/B testing doesn't look into the nuances of usability or productivity, it looks at easy-to-quantify metrics like conversion rates and money spent. These metrics rarely align with a better experience for the user (outside of like, prettier buttons and stuff), and instead tend to result in less-informative, less-agentic software (information and choice often distract from conversions!)
This is complete BS. I run hundreds of a/b tests each quarter and I specifically refuse to run the types of experiments you allude to. My a/b testing is all about helping users achieve the things (the outcomes) that they want to achieve by using our product in the first place. If we can help them do that, with more ease, then we are creating a better experience.
Perhaps you should just agree that, "not all a/b testing is the same".
Did you even read my whole comment? It's BS because he/she/they blanketed it without taking any nuance which i tried to do with my comment + an example!
Quote - "Speaking from work experience: A/B testing doesn't look into the nuances of usability or productivity, it looks at easy-to-quantify metrics like conversion rates and money spent"
This is an interesting example, and perhaps pushes part of the blame and dislike for A/B testing onto tech companies' incentives.
If you're building a tool to make life easier for the user, something that gives them a better experience is your optimal outcome. This seems like a scenario where A/B can produce a good outcome.
The challenge is when you throw in an ad-based revenue model, and the A/B testing is then optimized for the opposite (eyeball-hours, linear metres scrolled per session, ad spots passed, ads clicked) - engagement-based business models end up (I'd argue) A/B optimizing for the opposite of what their users want, to get them to spend longer doing a task they could have done quicker.
> The challenge is when you throw in an ad-based revenue model
The funny thing is - the ad-based revenue model is not the only possible variant. Last time I’ve checked Facebook’s profits per user were $7 per quarter, that is $28 a year. At the same time I am paying LiveJournal $25 a year for the ad-free version. Just taking my money looks like a much better model in many respects:
- less overhead: a lot of people doing these studies how to force me to look at something I do not want to look at will be free to do something more useful to the society;
- streamlined relationship between me and my publisher: in this model there is no advertiser who can say “I do not like these texts, no revenue for you”.
That’s why I prefer to pay for some Substack authors, like Matt Taibbi and Glen Greenwald, than to try to fish their texts for free amid some sea of “clever” advertising (hey AI testers, I bought this thing already, what’s the point of forcing it on me again and again?).
I kinda wish that Brave model (my money distributed between sites I visited) got more traction. It looks much more healthy.
There are much better methodologies for speeding up worker productivity than A/B testing. A/B testing is designed to extract information from people you can’t do more complicated tests such as eye tracking or motion studies with.
The major issue with A/B testing in the workplace is it causes confusion and slows people down when you change things. Which makes these tests really expensive even if they are seemingly easy to preform. So, I would call it useful but flawed.
As someone who’s run literally hundreds of A/B tests, many of them on the backs of UX research with users in the field, people have no idea what they want. The anecdata is a place to investigate, but never the end of the journey.
The fear with direct user research is that, unless you have a team and budget for getting enough of a sample, one-on-ones might not only be unhelpful but actively harmful if you implement something that solves that customers' problem but otherwise gets in the way for other customers.
I'm having a difficult time imagining a situation where people's actual productivity using a piece of software can be so easily measured. I'm sure it happens, but I think it's safe to say this is the exception to the rule when it comes to A/B testing
The specific test that did it for me, is that Facebook ran this experiment where they logged users out and then wouldn't let them log them back in despite the correct password, just to toy with them to see how long/hard they would keep trying to login, in order to see how addicted they were to Facebook.
I was in the "B" group, and felt so humiliated at how many times I tried to reset my password to get into Facebook.
Wow. That is insanely user hostile and borderline gaslighting/psychological torture. That is truly one of the most insane experiments I've ever hard of someone running.
It was a footnote in the wake of the main psychological experiments facebook ran on its users back in 2014*, which is overshadowing my searches for this particular detail.
I had this happen to me I think, and another case where half my links/screenshots would randomly get censored. Not just like random blogs/tweets/forums but even to super mainstream stuff like wikipedia/cnn/bbc.
There's also a neat sleight of hand here. Your inventor of the wheel surely tested multiple variants to optimize for the utility of his invention to the user. The A/B testing that's problematic is about optimizing taking advantage of the user. That doesn't lead to better experience, but the opposite. This is what's increasingly popular, and this is what people complain about or want to see banned.
Related: attention economy is predicated on bad user experience, because it makes money from friction.
There's a reason Tristan Harris called upon SV to avoid "A/B testing ourselves into the 'gradient descent of mankind'".
My qualms are not so much with the method as the morals that guide it. It's agnostic but when operationalized in a faulty moral framework can definitely lead to bad results.
> And without A/B testing, every product you use would be worse.
The primary goal of A/B testing is to see what's more profitable.
If that happens to result in better UI that's a side effect.
In fact, it could result in less usability (relevant to this conversation, it probably resulted in the frustrating "algorithm-based" timeline at FB/Twitter/etc).
> > The primary goal of A/B testing is to see what's more profitable.
Well, I think I'll provide a disagreeing opinion. :)
I assume this opinion probably comes from your past experiences, and I believe it is true in many cases. Since I'm not American and have never worked in an American corporate environment, I can't say what is true over there... but my experience in EU and Canada with A/A/B, A/B/C and typical A/B testing (as well as building such testing tools for others) was not like that.
For example, when building tutorials for users, profitability is far from being the primary objective. Same goes for building documentation, programming languages, open-source software, internal tooling and other such things.
Of course, I get that in the end, profitability is the primary goal of the company (with some exceptions). But I maintain that not all A/B tests have profitability as their primary goal, which makes the previous statement an incorrect generalization IMO.
A/B testing lead to the development of effective "dark patterns" in UI that trick users into doing things they don't want or don't understand, and then making it difficult to undo.
My thinking is that in this alternate universe where pi=3, circles (with diameter 2 * pi * r) will look like hexagons (which have diameter 2 * 3 * r), so wheels would have to be hexagonal.
Certainly, because it probably never existed as a function for wheel. Do you have any evidence that the "well duh" criteria wasn't used, and hex wheels show up in the archeological record?
> A/B testing isn't a new thing - I'm sure the inventor of the wheel experimented with different shapes, and the buyer of the hexagonal wheel probably didn't have the best user experience.
Consider yourself that inventor, would you A/B test hexagonal vs round?
That sounds like a clear no, you already know the answer to the hypothesized A/B test.
As for the other aspects they sound like great targets for testing within different use-cases, but I'm not sure why that'd be an A/B test as we think of them now.
> And without A/B testing, every product you use would be worse. Not only would it be less profitable, but it would also be harder to use, less useful, and less productive.
Do we live in the same universe? As far as I can tell, software keeps trending worse. Usability is terrible, options and settings keep getting moved around and hidden, software is less responsive than it used to be...
You are free to keep using MS-DOS for your computing needs...
But the majority of people choose to use a modern computer, presumably because they find it overall more useful than their old MS DOS computer and software.
Sure - there are gripes, but they must be outweighed by something pretty big for 99.99% of people to choose a new computer over a 30 year old one.
>And without A/B testing, every product you use would be worse.
Worse for whom? I feel like a lot of the A/B testing results in more revenue, a more addictive app, and less user satisfaction, because they're not testing for anything beneficial to the user, because at least with FB, you're not the customer, their advertisers are the customer.
This is so wrong in its conclusion, that its hard to know where to start. First, we should be clear that we are talking about involuntary, undisclosed A/B testing.
I have not experienced a product become better for the user as a result of involuntary A/B testing in my entire adult life.
Producers and consumer have both an adversarial relationship and a mutually beneficial relationship, and the distinction between these two is essentially the split between voluntary A/B tests and involuntary ones. In the adversarial component, the producer is trying to figure out how to extract more money from the consumer, without improving the product. Alternatively, (and equivalently), how to make the product cheaper, but also worse, in a way that yhe customer doesnt notice (with their wallet). A proactive version of the "market for lemons".
For instance, if you A/B test your cancellation process to minimize the number of people who cancel their subscriptions, you will almost certainly do something that makes you some additional money, and is also unambiguously evil.
Any A/B testing that is mutual benefit to consumers and producers can be done with consent, by volunteers. And the miniscule amount of scientific rigor you would lose by doing so is not worth the tremendous sacrifice we have seen in quality of consumables in the past 2 decades (probably longer, but i do not have the personal experience to go longer)
You might be compelled to describe involuntary A/B testing as a strategy for maximizing evil subject to the constraint that it be legal, but it often dips its toes into seeing what is illegal but still profitable, and is capable of fundamentally undermining our legal system and even our political system.
The technology has grown more powerful. The addition of computers that can optimize essentially arbitrary objective functions has serious existential implications for humanity.
A blanket ban on the practice, incurring the total dissolution of any corporate entity found guilty of the practice of involuntary A/B testing, would be a start.
well, for starters, there would have to have been a product that improved at all. Those are already rare enough that I can enumerate them, and in each of those instances involuntary A/B testing can be ruled out for other reasons.
When craigslist added the map that shows you where all of the people are offering the thing you are interested in. That was a very good change, but thats pretty far from how craigslist operates.
When dominos stopped serving hot glue on cardboard, its pretty easy to see how that didnt come about by furtive A/B testing. They were pretty confident people would like the new pizza more than the old pizza. So they told them about it. Boy did that work for dominos.
That actually speaks more generally to my point. If you're making a change that you think people will like, you tell them about it, because even if it turns out that they dont like it more, the fact that they thought they would and you did it generates quite a lot of good will for them.
UX is orthogonal to the thing most web-based software companies optimise for. Unless users are paying for the service, what the company cares about is user engagement, not user experience. It's not that every software company ever doesn't know what they're doing, it's just that what they're doing is _at best_ tangentially related to improving UX. It is clearly not in the user's best interest that they feel compelled to check Facebook frequently throughout the day, or spend hours scrolling through their feed. You can spin that as the company just improving the experience so that people want to keep using it, but fundamentally it is intentional psychological manipulation.
Within a university, research with human subjects is required to pass an ethical review before it is allowed to proceed. Given the scale and impact of the research conducted by Facebook on its users, it is entirely reasonable to hold them to the same standard.
> what they're doing is _at best_ tangentially related to improving UX . . . You can spin that as the company just improving the experience so that people want to keep using it, but fundamentally it is intentional psychological manipulation.
So we agree that ab testing is good for optimizing toward a numerical objective. You then seem to think that either:
A) There are simply no numerical objectives that correlate with good ux or
B) Every software company ever is optimizing toward perverse incentives by which they take more money from their users while making their products worse
It’s probably B that you believe, and this is such a myopic and paternalistic view. There are a couple cases where it’s a problem, eg cancellation flows. But this problem is orthogonal to AB testing (try cancelling your newspaper subscription in 1994). AB testing is mostly just trying to improve the rate at which people sign up or buy something, and in this case, your objection hinges on the hidden premise that people are idiots.
> Within a university, research with human subjects is required to pass an ethical review before it is allowed to proceed. Given the scale and impact of the research conducted by Facebook on its users, it is entirely reasonable to hold them to the same standard.
obviously "A/B testing" is not "hiring psychologists", because one is in the category of experimental methods the other one is about human resources
yet there are obvious connections. A/B testing is used to increase "conversion", which is profitability. which is the same fuckin' thing as addictiveness in case of a site where you pay with your eyeballs
That doesn't make any sense. You decide what they read. You would want to know if it's harming their mental state. Closing your eyes to the impact you have does not negate the impact
It isn’t just the one thing, it is a pattern where when given a choice between respected a sense of ethics and decency or taking more money, Facebook as an org has at every instance that is publically known, has taken the money. The high salaries seem to be justified not by their technical skill but their willingness to do what they are told for momey without regard to conscience.
Read the whistle blower report, witness the evolution from seeing content from your friends posted in less addictive chronological feed to addictive content your friends like in the internet sorted by addictive news. Hell, the site started as a PHP hack to creep on pretty women. They sold a bunch of data to foreign adversaries. For years, they let people sell ads to Nazis. They don’t give the people faced with the psychologically brutal jobs of moderation get benefits. They have been a platform for genocide and government surveillance.
I might be missing some examples of them missing some money to do the right thing, but nothing comes to mind.
A/B testing (which is getting users to respond/react to a UX event, and choosing which outcome is more suited to the business) is considerably different from "can we manipulate users up front, to perceive or react to things assertively and programmatically, even if against their interests?"
You have a good point. But let's not forget that the Nazi's and the Japanese used to do incredibly invasive medical tests on human beings in the name of "science". (Even the Americans have done political and medical experiments on their citizens, using the CIA, on African Americans and criminals). All these are condemned today by the scientific community because of it caused great harm (or even death) to the subjects who never gave their consent to such experiments. The psychological experiments conducted by FB on its users was equally bad because it looked to trigger emotions in the users (a useful feature for an advertising platform), some of which could cause users to go into depression. I don't know if the FB people conducting those experiments are aware that even mild depression causes great stress on an individual, and serious depression triggers suicidal impulses.
(This is not an attack on you or your otherwise valid point. Just a reminder that people should be mindful of their ethical obligations to get informed consent and not cause harm to others with their experiments).
I doubt that the mentioned 'psychological experiments' are just A/B testing. There is a very strong case to be made that Trump and Brexit would not have happened without Facebook, and those are just two examples.
A/B testing isn't a new thing - I'm sure the inventor of the wheel experimented with different shapes, and the buyer of the hexagonal wheel probably didn't have the best user experience.
Multiply that by the number of people in the world and the number of products people use, and A/B testing is really up there as possibly one of the most beneficial ideas ever.
I really don't understand those who claim it should be banned - I see no way that testing two different versions of a website with people who desire to use that website can bring sufficient harm to outweigh those massive benefits.