Hacker News new | ask | show | jobs
by joshu 5054 days ago
so how polite does one need to be? One hit per x seconds?
1 comments

If the /robots.txt does not mention a Crawl-delay, one page per 3 seconds is often a safe value. Of course this rather heavily depends on the site. In any case, if you have any specific need, always contact the people responsible for the site. I occasionaly run custom queries against the database on request, for example.