Hacker News new | ask | show | jobs
by xyzal 124 days ago
One would assume legit spiders obey robots.txt.
1 comments
lolc 124 days ago
This, to me, is the strongest argument to offer these slop generators. It provides an incentive to follow the robots.txt.
link
xyzal 123 days ago
Exactly. You disobey robots file => we'll make your crawl gain a net negative.
link