[tetsusaiga]

>In the end I decided the only real way to tell if what we were doing was working, was to turn it all off, for several months to see if there was a measurable difference.

What?

If this person genuinely thinks that there is no way to measure an ad's performance while it is running, they are either too ignorant to be in business, or they hired someone who lied to them.

[yojo]

If you’re full e-commerce the only metric that really matters is conversions. How do you know your ads aren’t just taking credit for customers who were going to buy anyway?

You can get great CTR and conversion rate running ads against your brand name. But if your SEO is sound you’re pretty much just canibalizing your organic traffic.

I’m not aware of a way to determine whether an ad just claimed credit for a customer it didn’t earn. I’m interested to learn though.

[jefftk]

> How do you know your ads aren’t just taking credit for customers who were going to buy anyway?

The word for this in the business is "incrementality", and there are several ways of measuring it. The simplest conceptually, is that you run two ads to two random groups of users: one for the product and one for something irrelevant like a charity. Then you compare conversions between the two groups.

(There are fancier ways to do it that don't require you to spend half your budget on an irrelevant ad, but that's the basic idea.)

[m3047]

The article suggests that they are comparing between two placements: the organic search placement and the paid placement (ad). I don't see any explanation why that is invalid. Indeed the literal meaning of incrementality would suggest a comparison of f(A) and f(A,B). Not quite "sampling with replacement", but if that's heresy to you you're welcome to quit reading now.

The protocol for doing so would involve studying what happens when the ad is present versus when it is not. The goal is conversion, scrupulously defined as people who click one or the other and subsequently purchase. Total conversion could go up, down, or stay the same. The only way the ad "wins" is total conversion increases, and even then maybe. If organic conversions went up when the ad was present you'd have a research problem! (The effect could be time-based, i.e. "awareness", or it could be a confounding externality.)

The protocol you suggest would seem to be removing the organic placement when the ad is present, that is: one or the other. On its face this sounds more "researchy" to me. Putting feasibility aside, it would plausibly be attractive to an advertising professional, but I would espect the customer to ask "why?" and I don't see the answer to that question. What's the motivation for this approach? I can see that it makes the advertising professional's contribution crystal clear, but why should the customer pay for it?

But hey I don't have 30 years of advertising experience, nor do I consider myself a statistician or machine learning expert. I do however have over 30 years of experience as an internet plumber and (more importantly here) data sous chef, so I've tasted a lot of ingredients in a lotta stews and have a solid grasp of experiment design and causality.

You work for the customer: consider that some avuncular advice.

[jefftk]

The problem with the approach in the article is that there are lots of reasons conversions can vary over time that are unrelated to what you're trying to study. If you have to use time to distinguish treatments, your best option is to alternate time periods (ex: one day on, one day off). But you can almost always run your two groups simultaneously, giving them different treatments, which allows you to eliminate the effect of timing noise.

[m3047]

Yes. Modulation not moderation. ;-)

That is what you're talking about, how slicing up your signal before transmission impacts your ability to receive it. Here's a Jupyter Notebook which will maybe make your head explode... I mean if you like math.

http://athena.m3047.net/pub/python/wiener-functions.ipynb

[whimsicalism]

I work in ads. You appear to be one of the few people on this thread who actually knows what they're talking about.

+1 to everything you've said.

[yobbo]

> is that you show two ads: one for the product and one for something irrelevant like a charity.

This is would be another experiment that does not test what you need to in order to demonstrate some sort of causality.

[jefftk]

Why doesn't that demonstrate causality? It is a randomized controlled trial.

[yobbo]

The control would be to not show ads. This was tried in the article, with time as the variable to partition the control groups.

[toast0]

Unless you control the ad platform, you can't make a 'no ads' control group; if you want to track the user, you've got to show them some ad; so an unrelated PSA is an ok option.

[jefftk]

Time doesn't work very well. There are lots of reasons why you will see different behavior at different times, so it really only works if you are trying to check something with a very large effect size.

Much better is to run your control group and experiment group in parallel, like I described above.

Is your objection that showing ads for something irrelevant could affect whether the user converts?

[whimsicalism]

People's behavior changes drastically with time. Showing no ad for your product seems like a sufficient control?

[yojo]

Thank you. I wasn’t aware of this, it makes perfect sense.

[tetsusaiga]

Thanks for adding this— I definitely failed in my explanation by not talking about how one runs such tests.

[Griffinsauce]

> one for the product and one for something irrelevant like a charity.

Those are extremely different, how does that prove anything?

[pedrosorio]

If you chose "not to show ads" to the control group, all the slots of the control group where you would have shown ads will have ads you don't control, including your competitors' ads.

If you show a neutral ad, you know exactly how much seeing your brand in ad drives revenue compared to seeing an irrelevant ad.

[m3047]

Hey Jefftk, this is a human dimension I wasn't acknowledging elsewhere. Theory is one thing and money is another. ("whiskey is for drinking, and water is for fighting over" or something like that -- Samuel Clemens)

So in the real world I bet this is what you're talking about: "but there are clickmonkeys out there" or words to that effect.

[edit:] But to be honest, I'm not sure it directly answers the question about entanglements between ad and no ad.

[jefftk]

The goal in showing an ad for something irrelevant is to have a control group: one where it is very unlikely that your interaction is going to affect their purchasing your product.

[tomrod]

Yaye for A/B testing!

[tetsusaiga]

I agree that only conversions matter, but you should be able to track whether a customer originated from an ad click vs. SEO, whether they clicked one ad and had 4 sessions since, etc.

While it's not a perfect system, you should know with a reasonable degree of accuracy how many dollars you get back for every dollar you put in to ad spend.

As far as the SEO traffic issue, this can also be accounted for once you have scaled your ads beyond the levels of what SEO could yield. At low budgets this can be more of an issue, at high budgets much less so.

[yojo]

I don’t know how counting sessions or source of click matters. You still don’t know whether the customer would have ultimately found you without the ad.

Put another way, the only way to tell if a customer will still find you without the ad is to not run the ad.

If you have sufficient traffic you might be able to measure this by turning your ad spend down rather than off and measure the bottom line impact. But other than gross spend changes there’s no way of telling for sure that the ads do anything.

[philote]

This is like the paid search results that come up when you search for a specific company name. The company's site is just below those paid results, but often the same site is in the paid results but they are further up so are more likely to be clicked.

[tetsusaiga]

> You still don’t know whether the customer would have ultimately found you without the ad.

Sure, maybe 100 customers would eventually find you. But if you buy the traffic, you can make them all find you on the same day.

> But other than gross spend changes there’s no way of telling for sure that the ads do anything.

I'll use the most fundamental example:

If someone clicks your ad and buys the product during that session, you know the ad worked. If you keep increasing your ad budget every day, and you are consistently returning $2 for every $1 you spend, you know it's working. Turn off the ads, and the revenue goes away. You'd be surprised how many people make their living doing this.

[thrwy_ywrht]

>If someone clicks your ad and buys the product during that session, you know the ad worked

This is not as clear as you make it seem. Let's say there's a hypothetical product that a consumer only buys once every 5 years. If someone clicks your ad and then immediately buys the product... what if they would have bought it anyway, tomorrow, or 5 minutes from now, without the ad? How can you test that counterfactual?

If I buy cat litter online once a month every month for 5 years by going to example.org/catlitter -- but then they decide to start advertising on facebook, so now I click the facebook ad once a month to buy cat litter from the same site, are the ads "working"?

[tetsusaiga]

The idea is because it's happening at scale. If you're getting hundreds or thousands of customers per day, the likelihood of this happening gets lower by some statistical proportion, especially if you don't have the organic exposure.

Also, definition of working: You make more than you spend, and if you stop the ads, you stop making as much.

That said, I'm not arguing for 100% accuracy either. It's certainly not. Simply that it's possible to be more accurate than not, which leads to profitability.

One more also- a comment above about how I failed to mention I'm not talking about paid search ads so much as other types like FB, YouTube, banners, etc.

[yobbo]

> [...] Turn off the ads, and the revenue goes away.

This is the experiment from the article. It showed that ads did not affect revenue.

[Closi]

> I agree that only conversions matter, but you should be able to track whether a customer originated from an ad click vs. SEO, whether they clicked one ad and had 4 sessions since, etc.

An ad-click could have been an organic click if the ad wasn't there though, which is where the complexity is.

The ad might have great conversion, but if the customer would have clicked an organic link that navigated to your site anyway then the ad is taking credit for an organic sale.

[tetsusaiga]

While you're not necessarily wrong, the distinction lies in how many people see you organically.

If I get 100 organic impressions/day, and then spend $N to get 500/day, I'm speeding the process up, and nearly guaranteeing to get in front of people who would never see me organically.

*Note: I'm largely not talking about paid search ads. Those are definitely an area where you can end up competing with yourself. Sorry to anyone who I replied to earlier, and wasn't clear enough on this with.

[Strom]

> you should be able to track whether a customer originated from an ad click vs. SEO

That's easy to track, but answers the wrong question. It answers whether the person clicked on the ad or the organic link. What we actually want to know is whether this person would convert even without seeing an ad. The only way to answer whether a person would still convert without seeing an ad is to make sure that they don't see an ad.

[tetsusaiga]

My major failure in this thread was not specifying from that start that I'm not so much talking about paid search ads (google ads), as much as I mean FB, YouTube, etc. That's the arena where you've really gotta question if you're competing with yourself.

On the flip side of that, most of HN seems to think purely in terms of paid search ads, when there are many other types of online advertising that exist, and that don't have this issue baked into them.

Such is the nature of internet dialogue, I guess lol.

[Strom]

That's a good point. It's a lot harder to cannibalize organic traffic with non-search ads.

[m3047]

Are you conflating a call to action on a channel you already control with advertising? Because in that case you might try f(A), f(A,B) and also f(B): could be a synergistic effect, or they could just click on anything.

[Nextgrid]

> whether they clicked one ad and had 4 sessions since, etc.

That's not trivial to do, both from a technical point of view (browsers - rightfully - fight these kinds of tracking attempts) but also legal (GDPR mandates that the customer opts into this kind of tracking but they have no incentive to do so).

[tetsusaiga]

You're right, this is actually a lot harder to recently, with the iOS updates that occurred a year or so ago. It's a space that evolves very quickly.

But this is a good place to point out that, yes - there is nuance to all of this - and I didn't really mean to turn it into a thesis on online advertising (lol) so much as to say:

It's still easy to be more accurate than "Turn it off and see what happens".

Even a semi-sophisticated media buyer is going in with a plan, and some method of measurement.

[m3047]

GDPR doesn't stop me from having ad1.example.com and ad2.example.com landing pages. Aspirationally perhaps, but technically no. I think this is different from fonts.gstatic.com in that it doesn't need to follow people around the internet and it's also not info necessarily going to a third party.

[dangerface]

> How do you know your ads aren’t just taking credit for customers who were going to buy anyway?

If you are doing advertising right you build a funnel from awareness through to conversion and track every part of it so you then a/b test your adverts and channels.

Getting advert channel fit is the key to success and can only be done by testing and measuring your adverts. The fact that op couldn't measure the effectiveness of their adverts shows they weren't doing this and their bad outcome should be 100% expected.

If you are just buying adverts in the hope you get more conversions then you are putting those adverts out to die. Burn your money you will probably get more eyes for doing that than untested adverts.

[whimsicalism]

This is now impossible to do on iOS devices.

[dangerface]

How is it impossible on iOS?

[whimsicalism]

Click attribution is basically not possible due to new Apple policy on tracking, unless you opt-in.

[boplicity]

> How do you know your ads aren’t just taking credit for customers who were going to buy anyway?

If you can't answer that question, then you're not running a proper advertising campaign. You're just throwing money around blindly. Plenty of people do have good answers to that question, though. Unfortunately, plenty of people don't.

[Strom]

> If you can't answer that question, then you're not running a proper advertising campaign.

What is this proper advertising campaign? Does it include turning off ads for terms that rank organically high anyway?

[boplicity]

> What is this proper advertising campaign

Well, for our business, proper advertising campaigns have opened new channels for growth, allowing us to reach new people, and build our email list significantly, with a predictable ROI. It's pretty easy to tell if an ad is working when you turn it on, and your email list growth rate instantly doubles -- and all the traffic is coming from a new traffic source. (Obviously, if we were to run Google ads on search traffic, we would check whether our organic search traffic dropped when we turned the ads on, but that's a different point.)

[Strom]

> It's pretty easy to tell if an ad is working when you turn it on, and your email list growth rate instantly doubles -- and all the traffic is coming from a new traffic source.

Maybe for a real tight definition of instantly, but generally this is not true. Imagine the following scenario:

1. You run an ad for ComapnyName.

2. An enthusiastic customer promotes your business in a local bar.

3. The bar attendees search for CompanyName and click on the first result.

Because you ran an ad for CompanyName, the first result is going to be your ad. You will see very nice ROI on that ad. What you won't see in any stats, is that these people would have probably found the organic link anyway, because they were already motivated and searching for CompanyName in particular.

> Obviously, if we were to run Google ads on search traffic, we would check whether our organic search traffic dropped when we turned the ads on, but that's a different point

That wouldn't help either in this imaginary scenario, because these bar attendees are a spike in traffic. Plus if the ad has been running for a longer period, then you won't have accurate organic search traffic stats anymore either, because it's already cannibalized.

[boplicity]

> Imagine the following scenario

"Imagination" is the core of the problem here. Plenty of people are imagining various scenarios, whereas people with successful advertising campaigns are just raking in the money -- no need for imagination.

I've turned off plenty of ads when they stopped working. I didn't need to turn them off to discover they stopped working.

Sure, you can imagine a dozen scenarios where an ad campaign doesn't work. Fortunately, I've managed to learn how to focus on the reality of the situation -- and have reaped the rewards in the process.

> you won't have accurate organic search traffic stats anymore either, because it's already cannibalized.

That' not true for us -- as I said before, a successful campaign for us brings in new sources of traffic from new marketing channels. One of the early lessons I learned was to not cannibalize what's already working. For example, we grew our Facebook Page (and email list) from scratch, when we had no Facebook traffic, by using Facebook advertising.

Operating under different brands -- even just for testing purposes, is a one another of the way we deal with this. Of course you can continue to imagine scenarios where we might be making mistakes. I do that as well -- it's called planning. Though, none of that really matters until money is spent (and made or lost).

[civilized]

One problem is the ad industry is quite good at finding ways to take credit for conversions that would have happened regardless, and analytics are all systematically biased in a way that conveniently maximizes the ability to do so.

The idea to simply turn off ads and see the effect is born of a healthy distrust of ad analytics industry bluster.

[josefresco]

> One problem is the ad industry is quite good at finding ways to take credit for conversions that would have happened regardless

This is a great counter-point when someone mentions "use conversion tracking!" Conversion tracking is great, but not if they store a cookie for 7/15/30 days and "award" the conversion to the ad, when the customer took a different and varying path to purchase. Sure the ad contributed "some" to the conversion, but not 100%.

[civilized]

I strongly suspect they sometimes show the ad and credit it for the conversion after the user shows initial interest in some specific product, i.e. when the user was already highly likely to buy before seeing the ad. This would explain why you're often flooded with ads for something after you search for it, go to its website, or purchase something there. (We've all had that experience of buying something and then being tailed by ads for that thing for weeks afterwards, even though no sane person would think we were going to buy that thing again in such a short time frame.)

A story to illustrate: there was once a pizza store that had two guys go out into the city to distribute promotional coupons. The coupons had codes on them so the business could attribute sales to each coupon distributor. John went out into the city and tried his best to drum up new business. Chad stood next to the door of the pizza place and handed a coupon to anyone who was walking in. After a month, 98% of the coupons used were from Chad. Chad got a big bonus and John was let go.

[LightHugger]

Companies that sell ads will do everything they can to convince you there is a measurable difference, but often while the metrics presented to the client are real, they do not always translate to real world impact.

There is no easy way to determine whether this is the case without just comparing when the ads are on and off and somehow dealing with the confounding variables, which isn't easy.

[bschne]

Well that works in some ways, but it can also be very hard to disentangle the actual causal impact of an ad campaign! Absent running an experiment, you don't really know if people would or would not have bought through some other channel without your campaign.

The most striking example is probably the experiment Tadelis convinced eBay to run on their brand-name ads, though the subtlety applies to less obvious cases of questionable ad spend too — https://faculty.haas.berkeley.edu/stadelis/Tadelis.pdf

[tetsusaiga]

I'll just reference my other replies to the first part here, but thank you for that link too. I'll have to give it a read when I have more time later today

[mbesto]

> If this person genuinely thinks that there is no way to measure an ad's performance while it is running, they are either too ignorant to be in business, or they hired someone who lied to them.

Doing a blackout month is a legitimate technique.

I've deeply studied ad performance metrics and they are NOT as conclusive as people might think. If they were, you could just keep increasing the amount of $ spent and the conversions would go up. Ok, not exactly, there is usually a diminishing returns aspect, but you get the point.

A great example of why this requires more analysis is branded vs non-branded search terms on AdWords. Let's say you spend $10k/month on the brand "Mattel" (aka branded) and $10k/month on "toys" (aka non-branded). Mattel likely gets you like a 20x ROAS because people are probably searching for a specific Mattel toy. It's also potentially likely that if you completely turned off branded search you would net the same results.

Ad attribution is NOTORIOUSLY difficult: https://www.optimizesmart.com/what-is-attribution-problem-in... so to pretend like "this guy is an idiot, he doesn't know what he's doing" is disingenuous.

[yobbo]

> this person genuinely thinks that there is no way to measure an ad's performance while

This is the case in many situations. The time between initial visit to site to conversion can be weeks, months, or years. Meanwhile, customers will visit site from various platforms. There is no way of attributing value to a single click, it needs to be approximated as an aggregate.

[gk1]

Don't pretend attribution is simple or solved. If that were true there wouldn't be dozens of attribution products, who are still far from perfect.

The Dunning Kruger effect is strong in marketing. The more you learn about marketing the more reason you have to be unsure.

[m3047]

_How would you measure this:_

Does this ad improve conversion compared to not having the ad at all?