Y
Hacker News
new
|
ask
|
show
|
jobs
by
catboybotnet
1028 days ago
Seconding. robots.txt is just a way of "asking nicely." If somebody wants to scrape, they're spoofing their UA and ignoring it. Can't do anything other than monitor the logs and ban IPs one by one.