Hacker News new | ask | show | jobs
by damonsauve 3081 days ago
Yeah, doesn't Google have multiple crawlers running with different missions? Ie, deep content crawlers and "fresh" content crawlers?