|
|
|
|
|
|
by greglindahl
3590 days ago
|
|
|
I was referring to an actual rate limit, not crawl-delay. For example, YouTube is pretty strict about rate limits: http://www.bing.com/search?q=%22We+have+been+receiving+a+lar... I agree that crawl-delay is rare, and often it's set too long so that it's impossible to fully crawl a site -- as if the webmaster set it up 10 years ago and never updated it as their site got faster and bigger. |
|
|