[drallison]

Another worthwhile article if you are building a crawler. http://www.michaelnielsen.org/ddi/how-to-crawl-a-quarter-bil...

[plantpark]

I've seen this article.Great article about distributed crawler. Mine just is a demo version of his.