[indymike]

> I fail to understand why Web Scraping isn't almost universally viewed as unethical and a terrible and nasty business practice.

Scraping is simply a way to get data. I used to run a team that was paid by large government contractors in the US to scrape their job posts from their career portals, and then deliver those posts via email, fax and snail mail to veteran's service officers near the job opening. It was required by regulation, and the only way to get the job data was to scrape.Many enterprise applicant tracking systems did not have a good way to automatically deliver that data or wanted $millions for that capability. Scraping was the best way and in some cases, the only way.

By the way, search engines like Google are scrape data and index it.