Hacker News new | ask | show | jobs
by mootothemax 3350 days ago
It depends on what and how you're trying to crawl, it's trivial to verify a "true" Googlebot using reverse DNS:

https://support.google.com/webmasters/answer/80553?hl=en

I know of a few sites that use this as the first step (of many!) to add bots to their "naughty" list.