[GlennS]

I liked this article, but I have two questions:

1. Is it definitely a good idea to build a separate data team, rather than embedding people with analytics knowledge in feature teams?

Is it possible to do the latter, but still have end up with a well-curated source-of-truth for your data?

2. Is A/B testing and driving your business by metrics really a good idea?

My (uninformed) impression is that data-driven is responsible for rather a lot of rot:

- Extremely irritating websites.

- Businesses ignoring important things because they can't measure them. (Financialisation, hand-in-hand with the MBA types the author decries.)

[dijksterhuis]

> Is it possible to do the latter, but still have end up with a well-curated source-of-truth for your data?

It's important to get the core centralised data infrastructure up and running (even if it's dirty af) as that helps with the bulk of the data work.

The oft quoted not completely true but kinda true statistic is that 70% of data work is finding, cleaning and storing the data. Analysis and modelling is the easy bit.

You could do it the other way around. Hire some data people in each team and get them to meet up every once in a while.

But I'd wager the central data stuff that makes everyone's life easier will get pushed back behind the "urgent" team work every time.

#ConwaysLaw

Edit: it's possible to do both btw. E.g. Have a bunch of centralised data engineers that do the heavy lifting stuff. With data scientist/analysts embedded in teams doing the fine grained modelling stuff. It's not a binary choice (once things are up and running).

> My (uninformed) impression is that data-driven is responsible for rather a lot of rot.

I agree! I was talking to someone else (not a tech head) the other week and realised why they hate tech so much... User interfaces that just... Don't work.

Showed him a terminal cli and he went nuts over it.

Then again, we're two kinda weird ye olde "back in my day" kinda people... So...

[dgb23]

Interesting. I'm a bit of a hybrid, CLI/GUI user. There are things that I find easier to to in a CLI (or with text in general) and things were a GUI is more natural.

CLIs are finicky and force you to think in terms of text, whether it is appropriate or not. GUIs can be more expressive and haptic, but are typically very idiosyncratic and can get in the way of things.

The data-driven approach to UI seems a bit crazy?

If I think about the problems of any UI, I think in terms of communication, intent, learning, psychology and aesthetics. All of those things are human to human or human to computer related issues.

I think data-driven (as in statistical data derived from user behavior) approaches are or can be useful in terms of "what" to present, prioritize and so on. But much less so on "how", because I think this should be based on experiences derived from direct interaction and needs to be induced by creativity.

And I mean creativity from both sides, the implementer and the user. One thing that CLIs generally do better is to provide composable tools within a adaptive and simple system (pipes, text etc.), whereas it is hard to impossible to let GUIs talk to eachother and compose them to a user tailored whole.

I think we should empower "non-technical" users with the freedoms and sound principles we have come to enjoy ourselves, instead of letting statistical data dominate their experience.

[wheelinsupial]

Is driving your business by the highest paid person’s opinion any different than driving it by A/B testing? I see those as two extreme end positions.

A/B testing can help you with optimizing existing processes for incremental improvement, but big bets, which can sometimes have data and sometimes don’t, help with step change improvements.

Even with big bets you need a way to show that it’s better than the previous way. Either by coming up with ways to cheaply test the hypothesis or committing to being “agile” (I hate that term) and continuing to iterate.

What is statistical significance anyways? If the p-value is 0.06 is that good enough? Practical significance is something that also needs to be accounted for.

If something can’t be measured, is there a way to find some proxy metric for it?

If not, then you can try to negotiate a pilot study of the problem and have specific criteria to determine success.

Just because something can’t be measured with existing processes doesn’t mean it can’t be measured at all.

For example, there were complaints about systems crashing and having intermittent behavior, and the claim was that’s affecting sales. Technology said nothing in our logs shows any issues, our service center shows no reporting of issues, so we think they are overreacting. We put a team together and went to several different locations to observe the process and get feedback. From the feedback we put together a data collection sheet and went back for a week to collect more data. That finally convinced the Tech team that it was a problem they needed to investigate. They went to the stores, determined it’s true, and amended logging to capture what’s truly going on.

[alzaeem]

I share the frustration with how many A/B testing driven development processes end up. Leads to a very iterative process with lots of small changes, rather than big bets. Also, trying to get statistical significance from iterative changes when you don’t have a ton of data is problematic.

[iamacyborg]

I think that’s just down to a lot of folks who think ab testing is the answer to every problem not necessarily having a background in maths or stats. I see it all the time in marketing teams where people’s are so conditioned to think of testing as the default that they don’t understand what they’re doing or why.

[tiagogm]

In my experience AB testing has a time and place - and that is after a certain level of traffic load and product/feature maturity and only to "validate" certain hypotheses.

For low volumes of traffic AB testing would takes ages to wield significant results and for products still maturing and shaping there is lot of "wisdom of crowds" data already available to help make decisions faster (ie: do you really need an AB test to know offering timely promotion to users helps convert?)

If you got a young product trying to grow, fast, it's a lot more effective to rely on experienced product people and off-the-shelf simple analytics to iterate quickly and to take some bets so one day you get to a point where AB testing "optimisations" starts to make sense.

It's a quite an interesting topic! I agree with you too - A/B test driven sites tends to culminate in terrible "cumulative experience" for users