|
|
|
|
|
|
by jwr
4933 days ago
|
|
|
If these scrapers abide by robots.txt (you do have a robots.txt, right?) and scrape only whatever you left publicly accessible, then I think you should work on your server performance, because if a scraper causes problems, you'll have much worse problems once whatever you've built becomes popular. Hunting down bots is a waste of time and effort better spent elsewhere. |
|
|