Hacker News new | ask | show | jobs
by GMali 5264 days ago
Can someone explain what exactly he meant by "throttling the google bot"?
1 comments

It's telling googlebot to slow down and not be so aggressive.

You can use google webmaster tools to dial down the crawl rate.

Some think you can use Crawl-delay: in robots.txt but I have not tested this (and it may very well not work).