[cj]

> The reason I say this though, is because if they knew how to run ads properly they would have been tracking their results from the start

Part of the problem is your brand keywords will typically show up as being one of your best converting keywords.

[VBprogrammer]

A phenomenon which reminds me of the canonical survivorship bias story. In WWII they conducted studies to determine where the bullet holes where on aircraft which returned from bombing sorties, in order to determine which parts of the aircraft required armour. It took a statistician to point out that they actual needed to armour those places where they rarely saw damage on returning aircraft, as those parts are most likely the parts where being hit caused the aircraft to not return at all.

Sometimes it requires a bit of a leap of imagination in order to resolve these things.

[tomrod]

Causal inference has made a lot of improvements since WWII, and "if" the advertising company knows what they are doing they run effective A/B or MAB testing; that said, statistical power is typically low because of insufficient sample size for individual companies.

You could pool all ads together, but since each advertising company is independent you get into all kinds of weird path dependencies.

While I wouldn't claim to be an adtech practitioner, I did at one point help a few F500 work through conceptual models of multitouch attribution and other statistical issues. These are very nontrivial issues -- proving advertising effectiveness is very difficult!

[staticautomatic]

A problem in what sense?

[jefftk]

It's a problem in that those keywords are some of the places where you are at least likely to be generating counterfactual conversions: most of that traffic was probably coming to anyway.