Hacker News new | ask | show | jobs
by golfer 139 days ago
As a site owner, how does one opt out of this, since it obviously ignores robots.txt?
1 comments
chadwebscraper 139 days ago
Shoot me your site and I can blacklist it
link
xnx 138 days ago
That's what robots.txt is for.
link