Hacker News new | ask | show | jobs
by perl4ever 2434 days ago
Has robots.txt been outlawed now? In what jurisdiction exactly?
2 comments
sjg007 2433 days ago
I don't believe that robots.txt has ever had the backing of law.
link
jjeaff 2433 days ago
Robots.txt doesn't restrict anything. It's just a request of what should and shouldn't be scraped by search engine spiders.
link