[frederi]

Just because you can do it without much effort doesn't mean less experienced people can. Crawling can be a barrier to some people.

[boyter]

To be honest a simple crawler is a very simple thing to write. If someone had issues getting that going I think they are going to have issues with the data volume anyway. LisaG answered why the 3 million data set though and I agree with the reasoning.