|
|
|
|
|
by jebblue
4655 days ago
|
|
>> The irony of this has always intrigued me: Google may crawl your servers, but under Google's policies, you may not crawl Google's servers. It looks like some of their site can be crawled and some not, that's how robots.txt has worked for a long time: http://www.google.com/robots.txt |
|
What are you suggesting?