[kilbuz]

Much of A/B testing is far deeper than the color of buttons.

How much faster can we process payments through provider A versus B in different countries around the world?

If we offer insurance after checkout, do we convert more than offering it before?

What ranking algorithm of skus leads to the highest conversion?

[williamcotton]

And what does any of that have to do with finding the right services or products for the right customers?

Does a customer care if you shave off 2% of the final cost or do they care about having world-class customer support?

Does a customer care if a product has a higher conversion rates or if it is the product they were looking for in the first place?

Does a customer care more about how long payment processing takes or do they care that it takes their local mobile payment app?

The only way to know is to talk to customers. Without doing so you’re just coloring a different variety of button.

[makeitdouble]

> Does a customer care if you shave off 2% of the final cost or do they care about having world-class customer support?

Times and times again, they'll tell you the latter and actually choose the former.

The same bargain as paying more to have no ads: people vocally push for no ads, some will ponny up the money, and the vast majority will make do with the ad supported model, while ad blocking or giving up on the service when they're fed up with it.

> Does a customer care more about how long payment processing takes or do they care that it takes their local mobile payment app?

I'm curious how you find the people to ask that ? If your current service doesn't provide support for the payment app, who would you ask if it was a deal breaker for them and refused to become your customer, not giving you anybof their information ?

[rightbyte]

> Times and times again, they'll tell you the latter and actually choose the former.

I don't think this is as true anymore.

There was some time between 1990 and 2015(ish?) where physical widget prices was falling faster then the quality decrease, software and computing hardware got better, where the quoted strategy made sense.

Nowadays you will get dropshipped crap (or any service sector equivalent) if you go for the lowest price.

[williamcotton]

I'm curious how you find the people to ask that?

Market research, talking to people in line at the post office, sending a nice personal email to an existing customer?

[tomnipotent]

We need a new "appeal to customer" logical fallacy. Talking to the customer is not a panacea for running a business, and the failed company graveyard is full of products that "delighted customers" but still couldn't cut it in the long wrong.

Part of running a business is having to explain to your boss how you spent millions of dollars, and building confidence that you're making sound decisions and not just shooting from the hip. Many times that will mean making decisions in the best interest of the company over the customer, and there's nothing wrong with that.

There's nothing perfect about A/B testing, and like any tool it can do both good and harm. But when I have to explain to my boss about how I'm spending their money, there's a limit to how much lip service I can play up about the customer journey before I have to put my money where my mouth is and demonstrate that I'm putting their cash to good use. A story that includes A/B testing along with qualitative customer research is better than a story that just includes one or the other.