Do any of you who scrape fear retaliation from the sites you scrape? Maybe you are violating a ToS or scraping copyrighted text, and they cut off your IP. Thoughts?
I think you have to take into consideration the TOS, copyright and also robots.txt. If you ignore these then its well within the site owners rights to do something about it - blocking you or further. We always look at the robots.txt file first and use that as our benchmark in terms of what they (the site) wish robots/crawlers to look at