Hacker News new | ask | show | jobs
by giltotherescue 5360 days ago
I run a site which triggers POSTs via Ajax for visitor tracking. We explicitly disable crawling of the Ajax directory via robots.txt and have not observed Googlebot crawling these URLs. Strange that Googlebot behaves differently for you.
1 comments

These don't seem like crawlers, but rather proxies requesting information on behalf of a user (perhaps with that user agent that is in the UA string). Those are "Google bots", but they aren't the Googlebot.