Hacker News new | ask | show | jobs
Robots.txt – Just Crawl It (nike.com)
2 points by bfoks 9 days ago