|
|
|
|
|
by corobo
4120 days ago
|
|
Rather than it hitting quota error pages would it be feasible to give Googlebot a 503 header back after a certain amount of pages? Setting a Retry-After header to the next day should let it know when to come back for more From https://plus.google.com/+PierreFar/posts/Gas8vjZ5fmB (Not sure how official this is but Pierre appears to work for Google) Primarily the section "2. Googlebot's crawling rate will drop when it sees a spike in 503 headers. This is unavoidable but as long as the blackout is only a transient event, it shouldn't cause any long-term problems and the crawl rate will recover fairly quickly to the pre-blackout rate. How fast depends on the site and it should be on the order of a few days." Edit: Looks like the over quota page is a 503. Couldn't hurt to do it early yourself, Googlebot will see it the same way whatever provides it the 503 |
|