[m3047]

The article suggests that they are comparing between two placements: the organic search placement and the paid placement (ad). I don't see any explanation why that is invalid. Indeed the literal meaning of incrementality would suggest a comparison of f(A) and f(A,B). Not quite "sampling with replacement", but if that's heresy to you you're welcome to quit reading now.

The protocol for doing so would involve studying what happens when the ad is present versus when it is not. The goal is conversion, scrupulously defined as people who click one or the other and subsequently purchase. Total conversion could go up, down, or stay the same. The only way the ad "wins" is total conversion increases, and even then maybe. If organic conversions went up when the ad was present you'd have a research problem! (The effect could be time-based, i.e. "awareness", or it could be a confounding externality.)

The protocol you suggest would seem to be removing the organic placement when the ad is present, that is: one or the other. On its face this sounds more "researchy" to me. Putting feasibility aside, it would plausibly be attractive to an advertising professional, but I would espect the customer to ask "why?" and I don't see the answer to that question. What's the motivation for this approach? I can see that it makes the advertising professional's contribution crystal clear, but why should the customer pay for it?

But hey I don't have 30 years of advertising experience, nor do I consider myself a statistician or machine learning expert. I do however have over 30 years of experience as an internet plumber and (more importantly here) data sous chef, so I've tasted a lot of ingredients in a lotta stews and have a solid grasp of experiment design and causality.

You work for the customer: consider that some avuncular advice.

[jefftk]

The problem with the approach in the article is that there are lots of reasons conversions can vary over time that are unrelated to what you're trying to study. If you have to use time to distinguish treatments, your best option is to alternate time periods (ex: one day on, one day off). But you can almost always run your two groups simultaneously, giving them different treatments, which allows you to eliminate the effect of timing noise.

[m3047]

Yes. Modulation not moderation. ;-)

That is what you're talking about, how slicing up your signal before transmission impacts your ability to receive it. Here's a Jupyter Notebook which will maybe make your head explode... I mean if you like math.

http://athena.m3047.net/pub/python/wiener-functions.ipynb