Hacker News new | ask | show | jobs
by troyvit 124 days ago
It looks like the tool lets anybody that robots.txt allows through. IOW it doesn't stop all public scraping, just the scraping you want to stop.