http://googleblog.blogspot.com/robots.txt
http://support.google.com/websearch/bin/answer.py?hl=en&...
There other ways of blocking crawlers other than robots.txt
http://support.google.com/websearch/bin/answer.py?hl=en&...
There other ways of blocking crawlers other than robots.txt