Hacker News new | ask | show | jobs
by xyzal 76 days ago
Exactly. You disobey robots file => we'll make your crawl gain a net negative.