Y
Hacker News
new
|
ask
|
show
|
jobs
by
erikano
4898 days ago
The Google crawlers, at least, will only consider a robots.txt on a subdomain valid for the subdomain it was served from.[1]
[1]:
https://developers.google.com/webmasters/control-crawl-index...