| You can easily tell Google not to index your site with a robots.txt -- although you probably don't want to. You can also tell Google to index your site more slowly, in Google WebMaster Tools, although if I remember right the setting expires every few months, and needs to be reset. The odd thing here IS that webmaster tools won't let him restrict the crawl rate, that's very odd. (Also, it would be nice if you could restrict crawl rate in robots.txt, not just webmaster tools). In the end though, if your business is going to depend on Google indexing it, then you don't really want to tell Google not to -- or, really even to tell it to index more slowly. But a robots.txt can be a temporary measure while you figure out what to do -- if you want Google to index your site, you've got to make your site able to stand up to googlebot traffic. Caching is often pretty helpful, and can help with your site's reliability and performance beyond googlebot issues. That's kind of just the way it is, right? If you want google index, you've got to be able to handle googlebot. Nothing too shocking here? Caching is definitely something to look into, that can improve the reliability and performance of your site beyond just dealing with googlebot. I guess the odd thing is that Webmaster Tools is not letting the author rate-limit. And it would be really nice if google defined and respected some extension to robots.txt to do it there. I guess you could always rate-limit google bot with your own firewall-ish tools, but it might make googlebot mad and you might get even less indexing than you wanted. Really, if your product's success depends on google indexing it, you don't want to slow it down anyway, except maybe as a temporary measure -- you're going to have to figure out how to handle it. People are usually complaining about how to make sure googlebot comes to _more_ of their site _more often_, not the reverse! |