[swarnie]

Allow? Deep down, do you think you have a choice?

Mechanisms might exist to make you think you have one, the same way copywrite should prevent millions of books being gobbled up by TheZuck but ultimately do you really have a choice?

Rules and laws don't exists for you.

[alt227]

Yes, Google advertises its crawler IP ranges and it is quite easy to keep track of this and block them. But only if you control the infrastructure that your site runs on of course.

[rolph]

stego your site, google sees the red herring version, intended users see the payload.

this has been done before, quite often, but toward ends morally askew.

[account42]

You think they'll stick to those IPs any more than robots.txt when it comes to that?

[alt227]

Yes I do, I also think they will continue identifying themselves in the user-agent header.

https://developers.google.com/crawling/docs/crawlers-fetcher...