Y
Hacker News
new
|
ask
|
show
|
jobs
by
arbuge
4475 days ago
You need to update those crawlers to enforce some hard limits on crawl depth, methinks, especially for low pagerank domains (if there's some easy way to estimate pagerank, i.e.).