[dopplesoldner]

Interesting idea, how would the framework scale to say ~1 million items?

[pdhanda]

Hi, we are currently in the process of building an enterprise-scale crawler. However, for the purpose of this beta test, we simply chose a few pages to index to validate the effectiveness of our idea.