[from-nibly]

I've never worked at a company where we were anywhere close to being able to A/B testing on customers. We just always shot from the hip. I think if you are doing that kind of testing may e you've run out of helpful ideas?

Talking to real customers and helping them solve real problems is really potent. And you can get more than just the color of a button. You can get the direction your company needs to go for months.

I think part of the problem is that science takes too long. It's like waiting for evolution to play out. You're company is at war with everything, entropy, the economy, your competition, the attention span of customers. Do you have time to science your way to success? Probably not. Do you have time to gamble on your intuition? Barely.

Collecting data isn't bad per se. But you should always be asking yourself if you are solving the right problems before you waste your time on it.

[traceroute66]

> Talking to real customers and helping them solve real problems is really potent.

This, this and THIS again !

Example case (of many I could cite) would be Transferwise.

They used to be good, but now they've denigrated into a quagmire. Could they be bothered to talk to their customers, or even just send round some box ticking surveys, they might find that out.

No amount of A/B testing, data lakes or other "data science" buzzwords is going to help them.

But no, instead its the same old story :

Rebranding from Transferwise to Wise because, well, I guess that's the usual shit companies do when they've run out of ideas (Aberdeen rebranding to Abrdn is another fine example from the financial sector).

Doing stuff worse because it benefits the business (read: increase margins) rather than the customer. Transfers take forever. Customer service is non existent.

Funnily enough it all seems to have started going downhill around the same time they floated on the stock market. Funny that !

Whilst I am aware that a company's strict legal definition is to put its shareholders first, it doesn't have to be that way, at least not in a blatant manner. Afterall, disgruntled customers don't do much good for shareholder's pockets.

[silverquiet]

I’m fairly sure there’s no legal requirement for a company to put shareholders first. I’ll sound a bit Marxist for a moment and say that we’ve just so internalized capitalist propaganda that we collectively seem to believe that now.

[traceroute66]

> I’m fairly sure there’s no legal requirement for a company to put shareholders first.

You're likely correct that there is no explicit legal requirement.

However (as I understand it), it stems from the implied requirement that derives from the fact that a company's directors have a fiduciary duty to act in good faith in the interests of the company.

People who agree with the implied requirement argue that "in the interests of the company" equates to "for the benefit of its members". And so you then ask yourself who are "its members" and that's where you end up at "its shareholders".

I believe in the jargon, this is referred to as "the common law approach of shareholder primacy".

Going back to the "legal requirement" front, there is, for example s172(1) of the Companies Act 2006[1], which starts by saying:

     "A director of a company must act in the way he considers, in good faith, would be most likely to promote the success of the company for the benefit of its members as a whole, and in doing so have regard (amongst other matters) to—"

So "must" is in the context of "benefit of its members as a whole", and a director is "only" required to "have regard" for other stakeholders that the legislation lists in (a)–(f). Its a bit of a word-salad, but effectively appears to re-enforce shareholder primacy.

[1]https://www.legislation.gov.uk/ukpga/2006/46/section/172

[Retric]

Firstly, financial benefits are only one aspect of possible benefits to stockholders. A company may consider reducing pollution a direct benefit to its owners as that improves their health. There’s quite a lot of freedom for things as cancer research for example benefits people beyond the financial incentives.

Anyway, the next sections makes it explicit that shareholder primacy isn’t required:

2)Where or to the extent that the purposes of the company consist of or include purposes other than the benefit of its members, subsection (1) has effect as if the reference to promoting the success of the company for the benefit of its members were to achieving those purposes.

(3)The duty imposed by this section has effect subject to any enactment or rule of law requiring directors, in certain circumstances, to consider or act in the interests of creditors of the company.

Shareholder primacy instead stems from shareholders being able to fire management.

[traceroute66]

Its clearly quite a technical topic.

But what I would say is, in relation to (2) that you highlight, that "purposes of a company" means that defined in its Articles of Association as created when it the company was formed (and as later amended if that is the case).

So I would argue (2) doesn't apply to the majority of companies, many of whom are likely operating off template Articles without expanded purpose definitions.

In relation to (3), interests of creditors, this was brought before the Supreme Court in recent history[1]. My reading of the summary of the judgement would suggest there is a relatively narrow window for being mandated vs "have regard", in particular:

"All members of the Court agree that AWA’s directors were not at the relevant time under a duty to consider, or to act in accordance with, the interests of creditors"

[1] https://www.supremecourt.uk/press-summary/uksc-2019-0046.htm...

[staticautomatic]

Maybe, but the business judgment rule is usually going to win if you want it to. Unsurprisingly, though, it’s rarely invoked to support doing something other than giving capital to shareholders.

[data-ottawa]

I hate the equivalence of data to A/B testing. If all your data people do is say a button should be yellow then you're not using data effectively.

You should be using data to invalidate your assumptions, separate the real from the perceived, and to draw those aha moments mentioned in the article. Then use that to prioritize and decide what’s worth iterating on and when its good enough to move on to bigger problems.

As the article says, data won’t tell you everything, which is why your data people need to also be product people, and not just sql monkeys or phds in a backroom doing analyses nobody will understand or read.

[fshbbdssbbgdd]

When I worked at a company that did a lot of AB testing, we had a lot of people whose main job was to interview users. This was an input in the development process. Then we tested whatever we came up with before releasing it.

I suspect there’s a disconnect here where you are talking about smaller, early stage companies. A lot of the time they don’t have the sample size to do proper AB testing, or the resources to do it properly, and they have less to lose. So shooting from the hip is more likely to be the only reasonable choice.

[disgruntledphd2]

AB testing works well in B2C, but is much harder to make work in B2B.

You can do it, but you can generally only test really large changes, and often if you have good customer communication you can pick up on what the change means with some interviews and showing the customer(s) what the new thing looks like.

This is generally much faster and cheaper (and I say this as someone who adores designing, running and analysing AB tests).

[makeitdouble]

> Talking to real customers and helping them solve real problems is really potent.

One of the many issues is you only get tot talk to customers willing to honestly talk to you.

That means you can't hear from potential new customers you wouldn't know were part of your market. You also don't hear from customers who would want to leave you but just haven't put it into words yet.

A/B testing helps get more insight into what customers actually do (and not what they tell you) and also get numbers on how big of an impact your changes have. The time to wait for the results is insignificant compared to the impact of ill changes in general.

[tonyarkles]

> One of the many issues is you only get tot talk to customers willing to honestly talk to you.

In the late 2000s I was part of a team that was developing some pretty incredible software to help chip designers manage the added complexity as features got smaller (context: our customers were freaking out about how hard it looked like 45nm was going to be). We did all of these customer satisfaction surveys and shit like that and got... some decent feedback but mostly just all rainbows and unicorns positive reviews.

Chip design software is complex and every customer of ours needed some custom integration, which is where my small group came in: the three of us were dual-degree EE/CS folks. We could sit down with the chip designers and understand their workflow and then go back to our hotel room at night and write the integration code to connect our tool with whatever bespoke workflow they had internally. All of that story leading up to the main point:

The feedback I got talking to random people outside in the smoking area was dramatically more valuable than anything we got from our customer surveys. This wasn't a strategy, I'd just go out for a smoke every hour or two to smoke and there'd usually be a couple of employees out there doing the same. "Hey, I don't recognize you, are you new?" "Oh, no, I'm here helping with the $X integration" "Oh! Hey so maybe you can help me then... in the latest release it looks like feature $X should be able to do $Y but I can't seem to get it to work..."

Pretty much every time I went outside I ended up learning something new, either an interesting way our software was being used or misused, or some other detail about how these guys' day-to-day workflow worked that we hadn't even thought of addressing.

We had some customers in Japan, too, where there's a an interesting social hierarchy when having business meetings. Me and the junior engineer across the table couldn't talk to each other directly in the meetings, all of the questions had to go through my manager, and a translator, and a senior manager on the other side of the table... in a big game of telephone even though we were in the same room. After the meeting I would usually go have a smoke and just happen to find the junior guy from the meeting doing the same. "You know, I do speak English... and have a few questions if you don't mind me asking directly" :D

While I can't recommend picking up a persistent nicotine addiction for doing better user research, I also can't say that I've ever encountered a more organic way to get really good unfiltered user feedback. Surveys, user studies, focus groups, etc... they're all decent tools to varying degrees but don't always get the level of honesty you can get out of someone sharing 5 minutes with you in the smoker's corner.

[devjab]

In my experience data often becomes the “decision argumentation” role which used to be filled by external consultants from E&Y and similar. Which is both good and bad, because data shows the past and that doesn’t necessarily predict the future.

Data can be very helpful though. We pull data from the public company records which show earnings to find possible investors. Then we combine that data with our sales data from HubSpot and Microsoft CRM (don’t ask me why we have both) as well as our internal sales systems. Which provides good data points for our sales department when deciding which potential investors to focus on, and shows them how much they’ve already “bothered” people. 10 years ago all of this was basically done by hand, now it’s mostly automated. Which sucks for the data researchers, but since the majority of those used to be unpaid students who now get to actually work on something more related to their studies, it’s mostly a win-win.

Where data doesn’t really help us is in marketing. Exactly because it’s showing us the past, and while that can be useful, it often hasn’t been very helpful in deciding how to do future campaigns. I imagine a lot of this is also true in other fields which produce content for human consumption. I guess in some areas it will be, but on most “creative” fields the data won’t necessarily show you what people will find “fun” or “interesting”. I think Hollywood, big gaming companies as mentioned in this article and others are sort of struggling with this. Thar is just my guess though as I only have experience with how our marketing department has come to the conclusion that while data is a good measurement of the success of various initiatives it’s not very useful in helping decide what sort of campaign to run next outside of which channels are the best focus, and even then, that also changes over time.

[internet101010]

The big tests that matter are always worth the wait. Making sure people don't screw it up by changing things mid-test can be difficult. Replacing the "most likely to be successful" with "most representative of potential rollout group" mindset of stakeholders isn't easy either.

Then you have people that try to get greedy. On more than one occasion I have designed a test where two variables change, results are great, rollout projections are great, the stakeholder attempts to do the rollout without changing the variable that creates incremental expense, and the rollout does not meet projections. Then they reluctantly do what they were supposed to do in the first place and everything is fine.

[coffeebeqn]

I was at a small games company that was way too small (in audience and headcount) to be A/B testing. They landed at a local maxima that was nowhere close to a sustainable business. The leadership was way too afraid to make actual decisions to actual problems and opted to change the color of the button for a few years until they ran out of money. ‘Twas a tragic waste of a team and money

[kilbuz]

Much of A/B testing is far deeper than the color of buttons.

How much faster can we process payments through provider A versus B in different countries around the world?

If we offer insurance after checkout, do we convert more than offering it before?

What ranking algorithm of skus leads to the highest conversion?

[williamcotton]

And what does any of that have to do with finding the right services or products for the right customers?

Does a customer care if you shave off 2% of the final cost or do they care about having world-class customer support?

Does a customer care if a product has a higher conversion rates or if it is the product they were looking for in the first place?

Does a customer care more about how long payment processing takes or do they care that it takes their local mobile payment app?

The only way to know is to talk to customers. Without doing so you’re just coloring a different variety of button.

[makeitdouble]

> Does a customer care if you shave off 2% of the final cost or do they care about having world-class customer support?

Times and times again, they'll tell you the latter and actually choose the former.

The same bargain as paying more to have no ads: people vocally push for no ads, some will ponny up the money, and the vast majority will make do with the ad supported model, while ad blocking or giving up on the service when they're fed up with it.

> Does a customer care more about how long payment processing takes or do they care that it takes their local mobile payment app?

I'm curious how you find the people to ask that ? If your current service doesn't provide support for the payment app, who would you ask if it was a deal breaker for them and refused to become your customer, not giving you anybof their information ?

[rightbyte]

> Times and times again, they'll tell you the latter and actually choose the former.

I don't think this is as true anymore.

There was some time between 1990 and 2015(ish?) where physical widget prices was falling faster then the quality decrease, software and computing hardware got better, where the quoted strategy made sense.

Nowadays you will get dropshipped crap (or any service sector equivalent) if you go for the lowest price.

[williamcotton]

I'm curious how you find the people to ask that?

Market research, talking to people in line at the post office, sending a nice personal email to an existing customer?

[tomnipotent]

We need a new "appeal to customer" logical fallacy. Talking to the customer is not a panacea for running a business, and the failed company graveyard is full of products that "delighted customers" but still couldn't cut it in the long wrong.

Part of running a business is having to explain to your boss how you spent millions of dollars, and building confidence that you're making sound decisions and not just shooting from the hip. Many times that will mean making decisions in the best interest of the company over the customer, and there's nothing wrong with that.

There's nothing perfect about A/B testing, and like any tool it can do both good and harm. But when I have to explain to my boss about how I'm spending their money, there's a limit to how much lip service I can play up about the customer journey before I have to put my money where my mouth is and demonstrate that I'm putting their cash to good use. A story that includes A/B testing along with qualitative customer research is better than a story that just includes one or the other.

[nonrandomstring]

The direct, old-fashioned approach of;

  - being real

  - talking to customers

  - actually listening to what people say

  - using intuition

  - cutting to the chase (big and meta problems first)

  - doing risky exploration for abductive reasoning 

is only as good as the nominal culture we're in. As the author says,

> I’m no longer a believer in decision-by-spreadsheet.

That's nice for you. Me neither. But every day we must interact with dull-headed data crunchers who set the pace and policy.

[hef19898]

All of the above needs some basis in reality, aka numbers. Otherwise it is just guess work.

[nonrandomstring]

What do you think is wrong with guess work?

[hef19898]

That you are as often wrong as you are right? If you have numbers, good ones, use them during decision finding.

[nonrandomstring]

50/50? For terrible guessers maybe.

For experts with tens of thousands of hours experience in a specialised field, with 40 years of case studies to extrapolate from?

Let's cast it in more relatable terms:

Such a person is, an enormous collection of data.

The day will come... soon, when people who "believe in technology" (in the very strong sense) will see no problem putting absolute trust in a neural network trained on exactly that same corpus of data.

A neural network is of course, a magnificent black box statistics machine.

And what are statistics machines trained on? Numbers. But they process and relate to them in a fuzzy way.

What is a spreadsheet and data analytics suite? Numbers.

Now your human specialist is going to outperform the numbers machine every time. But the human can often not introspect their ineffable knowledge (most expert knowledge is like that; which is why we developed the entire filed of expert systems to make it legible)

So if we choose to call such knowledge "feelings" of "guesswork" we're making a silly mistake. What does that even mean?

Neither can the neural network introspect. But we choose to label that ineffable knowledge as "calculation".

And so you invoke the magical properties of "NUmbers!" (did you mean real or imaginary ones :)

You see the error we fall into, giving two different labels to the same process only because of what hardware they execute on?

What I'd really like to talk about is the logical process of discovery called "abduction", but I fear I am rambling already :)

[hef19898]

I consider myself to be rather good in my field. Which is exactly why I take every bit of data I can get before I provide my opinion or decide something.

Every situation is different, facts change, so I have to evaluate my opinion each and every time (which is hownypu learn and bevome better). And the more data I have, the easier this is.

[carschno]

And because “science takes too long” (and is expensive, and tedious), people tend to fall back to pseudo-science. There is often no way to derive robust numbers from techniques like A/B testing, for instance because of confounding factors that are not measurable. Given that, I have regularly heard the argument that “these are the only numbers that we have”.

Relying on such numbers, however, is equivalent to falling back on intuition and gut feelings for decision-making (or worse), while believing that the decisions were based on numbers.

[Eridrus]

I think the discourse is converging on A/B testing being a big company thing, where I think it is very useful.

Not because it finds new knowledge, but because it keeps your product teams honest.

It's really easy to delude yourself and others about your project when your promotion is on the line, and A/B tests let you actually evaluate whether the change helped or not.

At small companies, you're not trying to find 2% effect sizes, anything that small is already a failure, so you don't need statistics to tell you what worked.