Hacker News new | ask | show | jobs
by lancer383 4112 days ago
Silly question, but wouldn't putting something like Crawl-delay in the robots.txt file (somewhat) help alleviate the situation (if it is respected)?

Or maybe even just block crawling of the entire site except for the homepage?

1 comments

Google doesn't respect crawl-delay, sadly. They rely on the Webmaster Tools setting, which is unavailable to me as I've described.