[agopaul]

A while ago while looking at the apache logs I noticed that the AdWords remarketing pixel does the same, it was trying to crawl private URLs that are only accessible to 'admins' that are not linked publicly. I'm not sure if this is still valid as I blocked by using robots.txt.

Also, the same crawler ignores the "User-agent: *" directive in the robots.txt file and you have to add specific rules for it: "User-agent: Adsbot-Google"