Hacker News new | ask | show | jobs
by dumbfounder 4931 days ago
Calling them "scrapers" implies they are doing something nefarious (stealing content). Robots.txt is for law abiding bots.
1 comments
ig1 4931 days ago
Not really. "Scraping" just refers to extracting data from a site using an automated method, it doesn't have any connotations about the motivation or acceptability of the process.
link