Hacker News new | ask | show | jobs
by bbernard 3354 days ago
Hey thanks for posting my article to HN. Glad that you found it helpful! :)
1 comments
carlmungz 3354 days ago
No worries. I'm into aggregation and scraping and your article had some good stuff in it. I've been scraping a site for a project of mine once every three minutes and I thought that was a bit much. Didn't realise it could be as low as once every 15 seconds.
link
bbernard 3354 days ago
The longer between your requests, the better.

If you generate a load greater than what a human would do, this might become problematic. A human wouldn't poke a website forever every 10-15 seconds.

The 10-15 seconds is more for web crawlers. Eventually, a crawler will run out of pages to crawl on a website, so it will stop sending requests to it.

Personally, I would stick to the 3 minutes delay. But it depends on what type of website you're scraping :)

link