This is an interesting example, and perhaps pushes part of the blame and dislike for A/B testing onto tech companies' incentives.
If you're building a tool to make life easier for the user, something that gives them a better experience is your optimal outcome. This seems like a scenario where A/B can produce a good outcome.
The challenge is when you throw in an ad-based revenue model, and the A/B testing is then optimized for the opposite (eyeball-hours, linear metres scrolled per session, ad spots passed, ads clicked) - engagement-based business models end up (I'd argue) A/B optimizing for the opposite of what their users want, to get them to spend longer doing a task they could have done quicker.
> The challenge is when you throw in an ad-based revenue model
The funny thing is - the ad-based revenue model is not the only possible variant. Last time I’ve checked Facebook’s profits per user were $7 per quarter, that is $28 a year. At the same time I am paying LiveJournal $25 a year for the ad-free version. Just taking my money looks like a much better model in many respects:
- less overhead: a lot of people doing these studies how to force me to look at something I do not want to look at will be free to do something more useful to the society;
- streamlined relationship between me and my publisher: in this model there is no advertiser who can say “I do not like these texts, no revenue for you”.
That’s why I prefer to pay for some Substack authors, like Matt Taibbi and Glen Greenwald, than to try to fish their texts for free amid some sea of “clever” advertising (hey AI testers, I bought this thing already, what’s the point of forcing it on me again and again?).
I kinda wish that Brave model (my money distributed between sites I visited) got more traction. It looks much more healthy.
There are much better methodologies for speeding up worker productivity than A/B testing. A/B testing is designed to extract information from people you can’t do more complicated tests such as eye tracking or motion studies with.
The major issue with A/B testing in the workplace is it causes confusion and slows people down when you change things. Which makes these tests really expensive even if they are seemingly easy to preform. So, I would call it useful but flawed.
As someone who’s run literally hundreds of A/B tests, many of them on the backs of UX research with users in the field, people have no idea what they want. The anecdata is a place to investigate, but never the end of the journey.
The fear with direct user research is that, unless you have a team and budget for getting enough of a sample, one-on-ones might not only be unhelpful but actively harmful if you implement something that solves that customers' problem but otherwise gets in the way for other customers.
I'm having a difficult time imagining a situation where people's actual productivity using a piece of software can be so easily measured. I'm sure it happens, but I think it's safe to say this is the exception to the rule when it comes to A/B testing
If you're building a tool to make life easier for the user, something that gives them a better experience is your optimal outcome. This seems like a scenario where A/B can produce a good outcome.
The challenge is when you throw in an ad-based revenue model, and the A/B testing is then optimized for the opposite (eyeball-hours, linear metres scrolled per session, ad spots passed, ads clicked) - engagement-based business models end up (I'd argue) A/B optimizing for the opposite of what their users want, to get them to spend longer doing a task they could have done quicker.