Hacker News new | ask | show | jobs
by lend000 3341 days ago
Unless you do it from a Google Cloud instance, that is.
1 comments

>Unless you do it from a Google Cloud instance, that is.

What's the reverse DNS for Google Cloud IPs? Google says to check that Googlebot's IP resolves to either a .google.com or .googlebot.com domain.

https://support.google.com/webmasters/answer/80553?hl=en

Couldn't you use GWT Mobilizer to scrape a site then index that?

Like this: http://i.imgur.com/ocR54Yq.jpg

Good point -- although it makes sense why it isn't frequently implemented. DNS lookups aren't cheap for this kind of thing.
It would be sufficient to let some requests come through from "Googlebot", and then deal with them (block, rate-limit, whatever) once the DNS check has been completed.
.googleusercontent.com