Hacker News new | ask | show | jobs
by jjeaff 2433 days ago
Robots.txt doesn't restrict anything. It's just a request of what should and shouldn't be scraped by search engine spiders.