Hacker News new | ask | show | jobs
by xyzal 77 days ago
One would assume legit spiders obey robots.txt.
1 comments
lolc 77 days ago
This, to me, is the strongest argument to offer these slop generators. It provides an incentive to follow the robots.txt.
link
xyzal 76 days ago
Exactly. You disobey robots file => we'll make your crawl gain a net negative.
link