Hacker News new | ask | show | jobs
by aequitas 294 days ago
Bad guys might download the robots.txt to find out the stuff they don't want them to crawl.