Hacker News new | ask | show | jobs
by troyvit 77 days ago
It looks like the tool lets anybody that robots.txt allows through. IOW it doesn't stop all public scraping, just the scraping you want to stop.