[greglindahl]

Search engine crawlers use adaptive politeness: start being very polite, and ramp up parallel fetches if the site responds quickly and has a lot of pages.

[stummjr]

That's kind of what Scrapy's AUTO_THROTTLE middleware does.