Y
Hacker News
new
|
ask
|
show
|
jobs
by
jjeaff
2433 days ago
Robots.txt doesn't restrict anything. It's just a request of what should and shouldn't be scraped by search engine spiders.