Hacker News new | ask | show | jobs
by erikano 4898 days ago
The Google crawlers, at least, will only consider a robots.txt on a subdomain valid for the subdomain it was served from.[1]

[1]: https://developers.google.com/webmasters/control-crawl-index...