I posted a comment about the web-crawling framework below so check that out for more info. Also, check out this if you'd like to know more about it @ http://bit.ly/1KiUmlS
'''
Crawlspace is a hosted crawling service that handles proxyification and everything for you behind the scenes. With Crawlspace, PledgeSociety was able to crawl and track every single creator on Patreon.
If you're interested in a service like this or you happen to do alot of crawling, we want to talk to you!
'''
Not my intention to be scummy. Just collecting emails and gauging interest. I had no intention to allow the public to use it but due to some others asking, I thought I'd set up a landing page for it. Appreciate the constructive feedback.