Hacker News new | ask | show | jobs
by catboybotnet 1028 days ago
Seconding. robots.txt is just a way of "asking nicely." If somebody wants to scrape, they're spoofing their UA and ignoring it. Can't do anything other than monitor the logs and ban IPs one by one.