Hacker News new | ask | show | jobs
by DamonHD 1007 days ago
Scraping hundreds of site seems mostly unethical.

My sites are periodically on the wrong end of scrapers, greedy by design or in error, occasionally needing to be manually blocked or even legals threatened.

Just because something can be done, done't mean it should. It also doesn't mean that you should make it easier. Any more than offering 'better' SPAM engines...

1 comments

Appreciate the input Damon, ethical concerns are definitely a consideration and we'll want to be respectful of mechanisms like robots.txt
Yawn, ignore that input. Scrape away!