[kevmo314]

This list mentions A/B testing a few times and it's worth noting that A/B testing is great but it's not free.

I've seen a nontrivial number of smart engineers get bogged down in wanting to A/B test everything that they spend more time building and maintaining the experiment framework than actually shipping more product and then realizing the A/B testing was useless because they only had a few hundred data points. Data-driven decisions are definitely valuable but you also have to recognize when you have no data to drive the decisions in the first place.

Overall, I agree with a lot of the list but I've seen that trap one too many times when people take the advice too superficially.

[simonw]

I think A/B testing is one of the most expensive ways of getting feedback on a product feature.

- You have to make good decisions about what you're going to test

- You have to build the feature twice

- You have to establish a statistically robust tracking mechanism. Using a vendor helps here, but you still need to correctly integrate with them.

- You have to test both versions of the feature AND the tracking and test selection mechanisms really well, because bugs in any of those invalidate the test

- You have to run it in production for several weeks (or you won't get statistically significant results) - and ensure it doesn't overlap with other tests in a way that could bias the results

- You'd better be good at statistics. I've seen plenty of A/B test results presented in ways that did not feel statistically sound to me.

... and after all of that, my experience is that a LOT of the tests you run don't show a statistically significant result one way or the other - so all of that effort really didn't teach you much that was useful.

The problem is that talking people out of running an A/B test is really hard! No-one ever got fired for suggesting an A/B test - it feels like the "safe" option.

Want to do something much cheaper than that which results in a much higher level of information? Run usability tests. Recruit 3-5 testers and watch them use your new feature over screen sharing and talk through what they're doing. This is an order of magnitude cheaper than A/B testing and will probably teach you a whole lot more.

[light_triad]

Some teams think they can A/B test their way to a great product. It can become a socially acceptable mechanism to avoid having opinions and reduce friction.

Steve Blank's quote about validating assumptions: "Lean was designed to inform the founders’ vision while they operated frugally at speed. It was not built as a focus group for consensus for those without deep convictions"

Is the Lean Startup Dead? (2018) https://medium.com/@sgblank/is-the-lean-startup-dead-71e0517...

Discussed on HN at the time: https://news.ycombinator.com/item?id=17917479

[eddythompson80]

Any sort of political/PR fallout in any organization can be greatly limited or eliminated if you just explain a change as an "experiment" rather than something deliberate.

"We were just running an experiment; we do lots of those. We'll stop that particular experiment. No harm no foul" is much more palatable than "We thought we'd make that change. We will revert it. Sorry about that".

With the former people think: "Those guys are always experimenting with new stuff. With experimentations comes hiccups, but experimentation is generally good"

With the later; now people would wanna know more about your decision-making process. How and why that decision was made. What were the underlying reasons? What was your end goal with such a change? Do you actually have a plan or are you just stumbling in the dark?

[whstl]

Something that is always a problem for me when doing A/B testing was that the C-Suite just reads the landing page of A/B testing tools where it says things like "Ridiculously easy A/B Testing" and they assume the tool is gonna do everything for them, including changing the page layout by simply adding some HTML in TagManager.

In my career I had the discussion to explain that this is not the case more times than it's appropriate.

A/B testing is possibly the most misunderstood tool in our business, and people underestimate even the effort it takes to do it wrong... let alone to do it right.

[bsder]

> You have to build the feature twice

Erm, isn't it three times, or am I missing something?

You have what you are currently doing (feature Null), feature A, and feature B.

Otherwise, you can't distinguish that the feature is what is causing the change as opposed to something else like "novelty" (favoring a new feature) or "familiarity" (favoring an old feature).

If all you have is "what you are currently doing" as "feature A" and "new thing" as "feature B", you're going to have a murderous time getting enough statistical power to get any real information.

[specialist]

Terrific criticism. Rings true. I've never done it (directly). Only watched teams and teammates get sucked into that void. Sadly, they eschewed most of your list.

I do know that Google strategically used A/B testing to determine the precise shade of corn flower blue for some UI widgets.

> Run usability tests.

Maybe the point of A/B testing is to avoid all that battle tested and validated '90s era process and methodological nonsense.

(As though a design could stumble thru a solution space to some local maximum. Like an optimization problem.)

I also kept thinking of that SQA cliché "You can't test your way to quality."

[porridgeraisin]

> You have to build the feature twice

Why though? Can't you have it dynamically look up whether the experiment is active for the current request and if so behave a certain way? And the place it looks up from can be updated however?

[stetrain]

But you have to implement and test both sides of that "if" statement, both behaviors. Thus "build the feature twice"

[simonw]

Right: you have to take responsibility for implementing (and testing and short-term maintaining) two branches.

[samstave]

The other issue with A/B is the imbued bias that comes from [TEAM] on [THING]

Meaning TEAM A has a want, and TEAM B has a want.. and its not about best feature its about "I told you so"

AB testing is great for sussing out use-flow-patterns, but oft may be used for ego (design team) testing...

[bluGill]

Most A/B tests should not be done in a production like way. Grab some post-it notes and and sketch out both A and B: then watch users work with it.

For a lot of what you want to know the above will give better information than a 100% polished A/B test in production. When people see a polished product they won't give you the same type of feedback as they will for an obvious quick sketch. The UI industry has gone wrong by making A/B in production too easy, even though the above has been known for many decades.

(A/B in production isn't all bad, but it is the last step, and often not worth it)

[marcosdumay]

You are pushing for bit of too much of name-misuse. A/B tests are something what the entire point is that you run them on the real world, and gather how real users react.

Design-time user testing has been a thing for much longer than A/B tests. They are a different thing.

I mean, your point stands. But you can't do A/B tests on anything that is not production, those your are recommending are a different kind of tests.

[bluGill]

I'll accept your definition correction. However I think my point still stands: there are better things than A/B testing to get the information you need.

[nightpool]

It probably helps that one of PostHog's core products is an A/B testing framework, so it's much easier for them to iterate on it internally for what they need to A/B PostHog. Even when you already have a best in class A/B testing framework though, I agree—A/B testing too much or waiting too long for "more data" to make decisions can slow down momentum badly for features that should be no-brainers.

[pkaler]

Agree!

Most orgs should just be shipping features. Before starting an Experiment Program teams should be brainstorming a portfolio of experiments. Just create a spreadsheet where the first column is a one-line hypothesis of the experiment. Eg. "Removing step X from the funnel will increase metric Y while reducing metric Z". And the RICE (Reach-Impact-Confidence-Estimation) score your portfolio.

If the team can't come up with a portfolio of 10s to 100s of experiments then the team should just be shipping stuff.

And then Experiment Buildout should be standardized. Have standardized XRD (Experiment Requirements Doc). Standardize Eligibility and Enrollment criteria. Which population sees this experiment? When do they see it? How do you test that bucketing is happening correctly? What events do analysts need? When do we do readouts?

That's just off the top of my head. Most orgs should just be shipping features.

[abxyz]

A/B testing is also high risk: a good test produces valuable data, a bad test produces harmful data. A company that does no testing is better off than a company that does bad testing. Many people treat product decisions made based on test results as unimpeachable, whereas they treat product decisions made on a hunch with a healthy skepticism that leads to better outcomes.

[baxtr]

You need to have a lot of traffic for it to make sense. And then the effect size needs to be big enough to notice. Not easy.