Hacker News new | ask | show | jobs
by _drimzy 2194 days ago
Why would a crawler want to crawl the same page every second (let alone 71 times a second). Even if I were Google, wouldn't crawling any page every few mins (or say every minute) be way more than enough for freshness?