|
|
|
|
|
|
by stummjr
3587 days ago
|
|
|
Scrapy is asynchronous, but it provides many settings that you can use to avoid DDoS a website, such as limiting the amount of simultaneous requests for each domain or IP address. And yes, crawling politely requires a bit of effort from both ends: the crawler and the website. |
|
|