[murtza]

Have you taken a look at the Scrapy framework for Python?

http://scrapy.org/

[jeffjia]

Thanks. I used beautiful soup for the parser, and actually have written a crawler framework for my scenario. But I was wondering whether there is any tool that could automate the selection of css selector or xpath.

[Larrikin]

I wrote a couple a few scrapers and found scrapy to be my best option