[none_to_remain]

Robots.txt is a hint to indexers that their index would be better off ignoring certain resources, not an authorization system

[erik_seaberg]

This. It was created as a warning of an infinite subgraph of generated hyperlinks, because crawlers would get permanently lost and DDoS sites.