Hacker News new | ask | show | jobs
by ihenriksen 4624 days ago
You should add some request-rate settings to your robots.txt file to prevent robots from crawling your web site to often. Here is an example: http://www.sparkledb.net/robots.txt