[pipihu]

The main use for robots.txt is to prevent crawling of infinite URL spaces: http://googlewebmastercentral.blogspot.com.br/2008/08/to-inf...

Alongside tagging links to such resources with nofollow.

[ashmud]

Back in the day, I would use httrack for offline web browsing, and these were a constant irritation.