|
|
|
|
|
|
by zasdffaa
1457 days ago
|
|
|
Crawling too fast from the same IP addy can get that IP blocked. Per 10 mins prob ok, per 1 minute may be too much depending on the volume being downloaded and the sensitivity (you mention saleable stuff in another comment so this might well make a difference). FYI. I did a reasonable amount of scraping myself. (err, crawling = scraping? or maybe they different things) Edit: tidy + clarify |
|
|