Hacker News new | ask | show | jobs
by randunel 442 days ago
Do user agents doing work for users need to respect robots.txt? If yes, does chrome?
1 comments
what 442 days ago
Any scraper is also a “user agent doing work for users”. Which ones should respect robots.tx?
link
randunel 442 days ago
Does the user agent fit the definition of a web crawler? If so, then observe robots.txt. This one does not, see https://en.m.wikipedia.org/wiki/Web_crawler
link