Y
Hacker News
new
|
ask
|
show
|
jobs
by
troyvit
77 days ago
It looks like the tool lets anybody that robots.txt allows through. IOW it doesn't stop all public scraping, just the scraping you want to stop.