Hacker News new | ask | show | jobs
by arbuge 4475 days ago
You need to update those crawlers to enforce some hard limits on crawl depth, methinks, especially for low pagerank domains (if there's some easy way to estimate pagerank, i.e.).