Hacker News new | ask | show | jobs
by gavinhoward 1038 days ago
Author here.

I wish I could, but I bet most would just ignore robots.txt.
1 comments
catboybotnet 1035 days ago
Seconding. robots.txt is just a way of "asking nicely." If somebody wants to scrape, they're spoofing their UA and ignoring it. Can't do anything other than monitor the logs and ban IPs one by one.
link