[cube00]

Sure, go join the realms of shady SEOs and malware, if I want to really stop you I'll know you're not coming from a Google IP range. https://www.incapsula.com/blog/was-that-really-a-google-bot-...

However, consider what your ultimate end game is, if it's a website you expect visitors to find through Google or the Play store, good luck once web masters start reporting your misbehaving "Googlebot" crawler.

[lend000]

Unless you do it from a Google Cloud instance, that is.

[mootothemax]

>Unless you do it from a Google Cloud instance, that is.

What's the reverse DNS for Google Cloud IPs? Google says to check that Googlebot's IP resolves to either a .google.com or .googlebot.com domain.

https://support.google.com/webmasters/answer/80553?hl=en

[milankragujevic]

Couldn't you use GWT Mobilizer to scrape a site then index that?

Like this: http://i.imgur.com/ocR54Yq.jpg

[lend000]

Good point -- although it makes sense why it isn't frequently implemented. DNS lookups aren't cheap for this kind of thing.

[Symbiote]

It would be sufficient to let some requests come through from "Googlebot", and then deal with them (block, rate-limit, whatever) once the DNS check has been completed.

[hrrsn]

.googleusercontent.com

[walshemj]

You know you have to be able to crawl your own or client sites using screaming frog or deep crawl as google to identify any crawl/ crawl budget issues.