|
|
|
|
|
|
by jakubbalada
3897 days ago
|
|
|
Yes, by default we respect robots.txt. There is a switch to disable it - on your own responsibility. We don't fully respect Crawl-delay, but minimum delay between requests for all our crawlers is set to 2000ms. We don't publish our IP ranges yet. |
|
|