Y
Hacker News
new
|
ask
|
show
|
jobs
by
ihenriksen
4624 days ago
You should add some request-rate settings to your robots.txt file to prevent robots from crawling your web site to often. Here is an example:
http://www.sparkledb.net/robots.txt