Hacker News new | ask | show | jobs
by jprete 531 days ago
I hope this is working out for you; the original article indicates that at least some of these crawlers move to innocuous user agent strings and change IPs if they get blocked or rate-limited.