[whimsicalism]

Here's a question I have about this experiment.

So the experimental setup is: you create an ad for your product, you create an ad for your charity, then you compare the populations of people who click on your ad for your product with the population of people who click on your ad for the charity and see if there is any difference in conversion rates between the two populations.

How do you ensure that the people who see/click on your product ad aren't already a population more likely to convert to your product than the population of people who see/click on your charity ad? Sure, you can target the same groups of people - but that only goes so far, the ML algorithms backing the ad selection process will still preferentially show your charity ad to people likely to click on charity ads and show your product ad to people likely to click on your product ad.

I am unfamiliar with how these experiments work on the advertiser side. Incrementality is easy to measure on the platform side if advertisers report conversion metrics to you.

[jefftk]

ML messing with you is an important consideration when you are trying to run this sort of experiment on a platform that you don't understand very well. The most simplest reliable way to run this experiment is to give the ad network an HTML creative that looks like:

    <script>
    var treatment = readTreatmentFromCookie();
    if (!treatment) {
      treatment = Math.random() < 0.5 ? CONTROL : EXPERIMENT;
      writeTreatmentToCookie(treatment);
    }

    if (treatment === EXPERIMENT) {
      showAd(PRODUCT);
    } else {
      showAd(CHARITY);
    }
    </script>

The creative is opaque to the network, which means it's not going to be able to do anything fancy like you're describing.

Many networks offer the ability to run a fully supported incrementality study, where they effectively use one company's ads as a control for another's, but this is a version of an experiment that you can run even if you don't trust the ad network at all.