|
|
|
|
|
by lancer383
4112 days ago
|
|
Silly question, but wouldn't putting something like Crawl-delay in the robots.txt file (somewhat) help alleviate the situation (if it is respected)? Or maybe even just block crawling of the entire site except for the homepage? |
|