[Jake232]

Not wanting to thread hijack, but just going to post an article I wrote a few years back as it covers a few other things that are still relevant; and often still gets referenced. May it'll help some people out in combination with OP's post.

http://jakeaustwick.me/python-web-scraping-resource/

[mdaniel]

I was surprised to not see Scrapy listed, but then I saw there were some comments about it - but seriously, doing by hand what Scrapy has spent years perfecting is highly suboptimal.

I guess the distinction is between whether one wants to just "toy around" or run the spider for-real.

[franciskim]

Nice post Jake!