|
|
|
|
|
|
by corobo
4117 days ago
|
|
|
Googlebot doesn't use the crawl delay line in robots.txt if that's what you're asking. I guess theoretically it's because you can normally change it in the webmasters interface. It's not really ignoring robots.txt either, as crawl delay is not an 'official' setting |
|
|