http://support.google.com/websearch/bin/answer.py?hl=en&...
There other ways of blocking crawlers other than robots.txt