[jefftk]

> How do you know your ads aren’t just taking credit for customers who were going to buy anyway?

The word for this in the business is "incrementality", and there are several ways of measuring it. The simplest conceptually, is that you run two ads to two random groups of users: one for the product and one for something irrelevant like a charity. Then you compare conversions between the two groups.

(There are fancier ways to do it that don't require you to spend half your budget on an irrelevant ad, but that's the basic idea.)

[m3047]

The article suggests that they are comparing between two placements: the organic search placement and the paid placement (ad). I don't see any explanation why that is invalid. Indeed the literal meaning of incrementality would suggest a comparison of f(A) and f(A,B). Not quite "sampling with replacement", but if that's heresy to you you're welcome to quit reading now.

The protocol for doing so would involve studying what happens when the ad is present versus when it is not. The goal is conversion, scrupulously defined as people who click one or the other and subsequently purchase. Total conversion could go up, down, or stay the same. The only way the ad "wins" is total conversion increases, and even then maybe. If organic conversions went up when the ad was present you'd have a research problem! (The effect could be time-based, i.e. "awareness", or it could be a confounding externality.)

The protocol you suggest would seem to be removing the organic placement when the ad is present, that is: one or the other. On its face this sounds more "researchy" to me. Putting feasibility aside, it would plausibly be attractive to an advertising professional, but I would espect the customer to ask "why?" and I don't see the answer to that question. What's the motivation for this approach? I can see that it makes the advertising professional's contribution crystal clear, but why should the customer pay for it?

But hey I don't have 30 years of advertising experience, nor do I consider myself a statistician or machine learning expert. I do however have over 30 years of experience as an internet plumber and (more importantly here) data sous chef, so I've tasted a lot of ingredients in a lotta stews and have a solid grasp of experiment design and causality.

You work for the customer: consider that some avuncular advice.

[jefftk]

The problem with the approach in the article is that there are lots of reasons conversions can vary over time that are unrelated to what you're trying to study. If you have to use time to distinguish treatments, your best option is to alternate time periods (ex: one day on, one day off). But you can almost always run your two groups simultaneously, giving them different treatments, which allows you to eliminate the effect of timing noise.

[m3047]

Yes. Modulation not moderation. ;-)

That is what you're talking about, how slicing up your signal before transmission impacts your ability to receive it. Here's a Jupyter Notebook which will maybe make your head explode... I mean if you like math.

http://athena.m3047.net/pub/python/wiener-functions.ipynb

[whimsicalism]

I work in ads. You appear to be one of the few people on this thread who actually knows what they're talking about.

+1 to everything you've said.

[yobbo]

> is that you show two ads: one for the product and one for something irrelevant like a charity.

This is would be another experiment that does not test what you need to in order to demonstrate some sort of causality.

[jefftk]

Why doesn't that demonstrate causality? It is a randomized controlled trial.

[yobbo]

The control would be to not show ads. This was tried in the article, with time as the variable to partition the control groups.

[toast0]

Unless you control the ad platform, you can't make a 'no ads' control group; if you want to track the user, you've got to show them some ad; so an unrelated PSA is an ok option.

[jefftk]

Time doesn't work very well. There are lots of reasons why you will see different behavior at different times, so it really only works if you are trying to check something with a very large effect size.

Much better is to run your control group and experiment group in parallel, like I described above.

Is your objection that showing ads for something irrelevant could affect whether the user converts?

[whimsicalism]

Here's a question I have about this experiment.

So the experimental setup is: you create an ad for your product, you create an ad for your charity, then you compare the populations of people who click on your ad for your product with the population of people who click on your ad for the charity and see if there is any difference in conversion rates between the two populations.

How do you ensure that the people who see/click on your product ad aren't already a population more likely to convert to your product than the population of people who see/click on your charity ad? Sure, you can target the same groups of people - but that only goes so far, the ML algorithms backing the ad selection process will still preferentially show your charity ad to people likely to click on charity ads and show your product ad to people likely to click on your product ad.

I am unfamiliar with how these experiments work on the advertiser side. Incrementality is easy to measure on the platform side if advertisers report conversion metrics to you.

[jefftk]

ML messing with you is an important consideration when you are trying to run this sort of experiment on a platform that you don't understand very well. The most simplest reliable way to run this experiment is to give the ad network an HTML creative that looks like:

    <script>
    var treatment = readTreatmentFromCookie();
    if (!treatment) {
      treatment = Math.random() < 0.5 ? CONTROL : EXPERIMENT;
      writeTreatmentToCookie(treatment);
    }

    if (treatment === EXPERIMENT) {
      showAd(PRODUCT);
    } else {
      showAd(CHARITY);
    }
    </script>

The creative is opaque to the network, which means it's not going to be able to do anything fancy like you're describing.

Many networks offer the ability to run a fully supported incrementality study, where they effectively use one company's ads as a control for another's, but this is a version of an experiment that you can run even if you don't trust the ad network at all.

[whimsicalism]

People's behavior changes drastically with time. Showing no ad for your product seems like a sufficient control?

[tomrod]

But then you have no control measure in most platforms, as you cannot track them?

[whimsicalism]

Attribution is not my area of expertise. But could you not track them to see if they convert? This is possible both on the advertiser side and the platform side, provided the advertiser reports conversions to the platform.

[yojo]

Thank you. I wasn’t aware of this, it makes perfect sense.

[tetsusaiga]

Thanks for adding this— I definitely failed in my explanation by not talking about how one runs such tests.

[Griffinsauce]

> one for the product and one for something irrelevant like a charity.

Those are extremely different, how does that prove anything?

[pedrosorio]

If you chose "not to show ads" to the control group, all the slots of the control group where you would have shown ads will have ads you don't control, including your competitors' ads.

If you show a neutral ad, you know exactly how much seeing your brand in ad drives revenue compared to seeing an irrelevant ad.

[m3047]

Hey Jefftk, this is a human dimension I wasn't acknowledging elsewhere. Theory is one thing and money is another. ("whiskey is for drinking, and water is for fighting over" or something like that -- Samuel Clemens)

So in the real world I bet this is what you're talking about: "but there are clickmonkeys out there" or words to that effect.

[edit:] But to be honest, I'm not sure it directly answers the question about entanglements between ad and no ad.

[jefftk]

The goal in showing an ad for something irrelevant is to have a control group: one where it is very unlikely that your interaction is going to affect their purchasing your product.

[tomrod]

Yaye for A/B testing!