[tetsusaiga]

I appreciate your thoughtful criticism of this.

The reason I say this though, is because if they knew how to run ads properly they would have been tracking their results from the start, and would have known much earlier whether they were getting a return on their investment.

If your only way of measuring advertising results is to "turn it off", you're just flying blind, and one can't expect success with a (lack of) strategy like that.

That said, you are not wrong that a meaningful percentage of advertising is being run with similarly insufficient statistical power, and to that I would say those businesses are also incorrect, for the most part. I delineate because at some point, say when you're Apple or Microsoft, you are so big that "brand awareness" advertising takes over performance advertising. For the most part though, I'd say those aren't the types of businesses we are discussing in a context like this one.

[Retric]

Tracking advertising effectiveness is ridiculously difficult and multiple people inside and outside your company are incentivized to overstate impact.

Statistical power for example assumes independence which can be very difficult. Great you spend X million to convince people to buy an AC in March, did you actually benefit or would those same customers want an AC as soon as the first heat wave hit? Spreading demand can be useful, but it’s also really easy to to draw false conclusions from statistics if you don’t understand the domain.

And that’s just one of the many pitfalls involved.

[tomrod]

You hit the nail on the head, for the record. Thank you!

[tomrod]

How do you know any strategy is attributable to success or failure without testing it?

Pre/post analysis may be temporally correlated but this isn't proof because you haven't captured a baseline comparison.

A/B and MAB testing are helpful but not magic bullets.

Shapley values (marginal impact) is a nice mathematical outcome to have for multitouch attribution but as usually implemented is only a single statistic and can be a fluke without additional testing.

[cj]

> The reason I say this though, is because if they knew how to run ads properly they would have been tracking their results from the start

Part of the problem is your brand keywords will typically show up as being one of your best converting keywords.

[VBprogrammer]

A phenomenon which reminds me of the canonical survivorship bias story. In WWII they conducted studies to determine where the bullet holes where on aircraft which returned from bombing sorties, in order to determine which parts of the aircraft required armour. It took a statistician to point out that they actual needed to armour those places where they rarely saw damage on returning aircraft, as those parts are most likely the parts where being hit caused the aircraft to not return at all.

Sometimes it requires a bit of a leap of imagination in order to resolve these things.

[tomrod]

Causal inference has made a lot of improvements since WWII, and "if" the advertising company knows what they are doing they run effective A/B or MAB testing; that said, statistical power is typically low because of insufficient sample size for individual companies.

You could pool all ads together, but since each advertising company is independent you get into all kinds of weird path dependencies.

While I wouldn't claim to be an adtech practitioner, I did at one point help a few F500 work through conceptual models of multitouch attribution and other statistical issues. These are very nontrivial issues -- proving advertising effectiveness is very difficult!

[staticautomatic]

A problem in what sense?

[jefftk]

It's a problem in that those keywords are some of the places where you are at least likely to be generating counterfactual conversions: most of that traffic was probably coming to anyway.