Hacker News new | ask | show | jobs
by corobo 4120 days ago
Rather than it hitting quota error pages would it be feasible to give Googlebot a 503 header back after a certain amount of pages? Setting a Retry-After header to the next day should let it know when to come back for more

From https://plus.google.com/+PierreFar/posts/Gas8vjZ5fmB (Not sure how official this is but Pierre appears to work for Google)

Primarily the section

"2. Googlebot's crawling rate will drop when it sees a spike in 503 headers. This is unavoidable but as long as the blackout is only a transient event, it shouldn't cause any long-term problems and the crawl rate will recover fairly quickly to the pre-blackout rate. How fast depends on the site and it should be on the order of a few days."

Edit: Looks like the over quota page is a 503. Couldn't hurt to do it early yourself, Googlebot will see it the same way whatever provides it the 503