[avereveard]

the problem is the hand wavy "user research"

if you have done that properly, why ab testing? if you did that improperly, why bother?

ab testing moves from an hypotesis, because ab testing is done to inform a bayesian analysis to identify causes.

if one knows already that the reason is 'button not visible enough' ab testing is almost pointless.

not entirely pointless, because you can still do ab testing to validate that the change is in the right direction, but investing developer time for production quality code and risking business to just validate something one already knows seems crazy compared to just ask a focus group.

when you are unsure about the answer, that's when investing in ab testing to discovery makes the most sense.

[tomnipotent]

> ab testing is almost pointless

Except you can never be certain that the changes made were impactful in the direction you're hoping unless you measure it. Otherwise it's just wishful thinking.

[avereveard]

I didn't say anything to the contrary, the quotation is losing all the context.

but if you want to verify hipotesis and control for confounding factor, the ab test needs to be part of a baesyan analysis, if you're doing that, why also pay for the priori research?

by going down the path of user research > production quality release > validation of the hypotesis you are basically paying research twice and paying development once regardless of wether the testing is succesful or not.

it's more efficient to either use bayesian hypotesis + ab testing for research (so pay development once per hypotesis, collect evidence and steer into the direction the evidence points to) or use user research over a set of POCs (pay research once per hypotesis, develop in the direction that research points to)

if your research need validation, you paid for a research you might not need. if you start research knowing the priory (the user doens't see the button) you're not actually doing research, you're just gold plating a hunch, then why pay for research, just skip to the testing phase. if you want to research from the users, you do ab testing, but again, not against a hunch, but against a set of hypotesis, so you can eliminate confounding factors and narrow down the confidence interval.