[xavian]

I maintain web crawlers for a large internet portal and this stuff is a pain enough to deal with when people unintentionally make their websites recursive, much less INTENTIONALLY. facepalm At least google crawler had issues as well. Thanks a lot, troll. ;P

[arbuge]

You need to update those crawlers to enforce some hard limits on crawl depth, methinks, especially for low pagerank domains (if there's some easy way to estimate pagerank, i.e.).