Y
Hacker News
new
|
ask
|
show
|
jobs
by
_drimzy
2194 days ago
Why would a crawler want to crawl the same page every second (let alone 71 times a second). Even if I were Google, wouldn't crawling any page every few mins (or say every minute) be way more than enough for freshness?