Hacker News new | ask | show | jobs
by blueskin_ 4117 days ago
>Your site has been assigned special crawl rate settings. You will not be able to change the crawl rate.

That's the messed up part. I guess the question is, does robots.txt override that or not? If it does, fine. If not, all you need to do is make a few "google ignores robots.txt" posts and the problem solves itself.

1 comments

Googlebot doesn't use the crawl delay line in robots.txt if that's what you're asking. I guess theoretically it's because you can normally change it in the webmasters interface.

It's not really ignoring robots.txt either, as crawl delay is not an 'official' setting