[rprameshwor]

I'm interested to know more about the tools/techniques you used to scrap the sites and clean the results. I am working on scraping contents from a couple of sites and it seems to be a pain, probably because i don't have lot of experience around it.

[wasi0013]

If you are familiar with python then try scrapy[1]. You can also scrape websites using beautifulsoup4[2]

[1] https://scrapy.org/ [2] https://pypi.python.org/pypi/beautifulsoup4

[codefined]

We actually used Node.JS' request module, combined with some NLP (using natural) in order to pick out the main content. This worked pretty well, but for our purposes we didn't need it to be perfect because anything like headers would be removed when we processed the content (not being full sentences).